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Chapter 1; Introduction 

The Measures of Success (MoS) project set out to develop, implement, and present the results of an 
evaluation model that could assess longer-term outcomes of workplace education programs for 
increasing literacy and Essential Skills. The model was tested at 18 work sites in Manitoba and Nova 
Scotia and included training programs that have been offered to employers over a number of years as 
well as several first time initiatives. Funded by Human Resources and Skills Development Canada's 
(HRSDC) Office of Literacy and Essential Skills (OLES), the project was managed by the Centre for 
Literacy of Quebec in partnership with Workplace Education Manitoba and the Nova Scotia Department 
of Labour and Advanced Education. 

The Social Research and Demonstration Corporation [SRDC) developed an evaluation approach for the 
Measures of Success project through a series of consultations with key stakeholders, subject-matter 
experts, and an extensive review of relevant literature and prior case studies. The proposed evaluation 
strategy included the following components: 

1. a recommended methodology for capturing outcomes of literacy and Essential Skills training based 
on a theory-driven multi-site case study approach; 

2. a rich evaluation framework that specifies a range of contextual variables, mediating factors, and 
outcomes of training at the individual and business level; and 

3. a series of survey instruments and protocols to collect the required data to support the evaluation. 

The project design report (SRDC, 2010) described in detail the design and development of this 
evaluation strategy. 

An earlier implementation report submitted to the project's Steering Committee (SRDC, 2012] 
described the process of recruitment and training delivery and provided a baseline profile of the 
participant sample and pre-training conditions within each of the 18 participating business. The 
research framework and baseline data were used to illustrate alternative hypotheses about how 
various conditions may influence the likelihood of observing post-training outcomes in different 
contexts. That report also offered early observations about data quality. 

1.0 Purpose of the Report 

This report builds on these early findings through an analysis of post-training outcomes. The focus is on 
the longer-term results from the final follow-up at 6-months after training onset. Some contrasts are 
made with the early 3-month data when they help inform the findings. The Final Report has four 
primary objectives: 

1. To review the design and implementation of the Measures of Success project including a 
description of the training provided and a profile of the participating businesses and workers 

2. To assess changes in key participant and business outcomes, from baseline to 6-months after the 
start of training, which may have been affected by the LES (Literacy and Essential Skills) training 
intervention 
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3. To explore the relationship between key contextual factors, the characteristics of participants and 
businesses, and the observed outcomes in order to offer further evidence of their link to training 
and to better understand the conditions that influence training success 

4. To make recommendations for revisions to the research tools to make them more reliable 
instruments that capture outcomes of LES training with less burden on participants and employers. 

1.1 Scope and Structure of the Report 

The report is organized into four sections corresponding to each of the central objectives. 

Part 1 reviews the design and implementation of the project including a description of the evaluation 
model in Chapter 2, the recruitment and training process in Chapter 3, and a baseline profile of the 
participating businesses and workers in Chapter 4. 

Part 2 explores changes in key participant and business outcomes from baseline to 6-months. 

Outcomes are organized according to the research framework. Chapter 5 considers the important 
intermediate outcomes associated with human, psychological, and social capital. Chapter 6 presents 
changes in everyday practices, workplace performance, and the longer-term financial and non-financial 
outcomes including health and well-being at the participant level and various tangible and intangible 
outcomes at the business level. 

Part 3 reviews the analysis of the explanatory factors that can help explain outcomes related to 
training. Chapter 7 discusses each of the factors according to the framework including participant 
demographics, firm characteristics, implementation factors such as training hours, method of delivery, 
and several additional mediating factors such as baseline levels of social and psychological capital. 
Chapter 8 presents a series of business contrasts that illustrate how the framework and contextual 
explanatory factors help us understand differences in outcomes across firms. 

Part 4 recommends ways to refine the research instruments to improve their reliability and reduce 
their research burden [i.e. the amount of time required to complete the instruments). Chapter 9 
reviews the performance of each of the key measures in terms of their response rates, distributional 
qualities, and in their utility in detecting changes or as explanatory factors. Chapter 10 presents a 
concluding summary of key findings from the project while a series of Appendices provide the revised 
research instruments along with guidelines for their use. 
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Chapter 2; Project Design 

2.0 Key Research Questions 

The Measures of Success project addressed three major research questions: 

1. What are the long-term outcomes of workplace LES initiatives in Manitoba and Nova Scotia on the 
participants, workplaces, and companies (where longer term is defined as six months)? 

2. What is a valid and reliable model for evaluating longer-term outcomes of workplace LES 
initiatives? What are the appropriate measures to be used? 

3. What are effective and efficient ways to provide workplace LES initiatives to maximize positive 
long-term outcomes? 

2.1 Theory of Change and the Research Framework 

The methodology used in the Measures of Success project for evaluating longer-term outcomes of the 
Essential Skills training interventions is based on a theory-driven multi-site case study. Specifically, it 
utilizes a theory of change approach that emphasizes the construction of a rich program logic model, 
which outlines all the implicit assumptions for how an intervention is expected to produce a specific 
result and the underlying steps that would lead there. Logic models describe logical linkages among 
program resources, activities, and outcomes - it is a narrative or graphical depiction of a Theory of 
Change. It clarifies how the change process will unfold, and places attention on the intermediate 
changes that need to occur in order for long-term outcomes to be reached. Importantly a theory of 
change is most effective when it is developed through an iterative process that begins with a review 
and consolidation of prior knowledge on a topic, followed by a consultation with key stakeholders who 
have unique knowledge about the possible effects of a program and its underlying assumptions. 

Figure 1 presents this project's logic model. It should be read from the top, starting with the Essential 
Skills training and the learning process and ending with the longer term outcomes. Between the 
learning process and the long-term outcomes are intermediate outcomes, many of which are 
hypothesized to influence the relationship between the workplace LES training process and the long- 
term financial and non-financial outcomes that individuals, businesses, and governments care about. 
Surrounding the model are some of the contextual factors that must be considered when capturing 
outcomes of adult learning. These factors may affect outcomes at each stage in the process. 

The logic model is based on a theory of change that was developed from a review of the adult learning 
literature including a model developed in New Zealand (see Folinsbee and Hayes, 2010), a sample of 
Organizational Needs Assessments (ONAs) from each province, and a series of interviews with key 
project stakeholders and practitioners in each province. It includes a range of possible outcomes of LES 
training, supported by evidence of varying degrees of quality. Some outcomes may be pervasive and of 
a great magnitude, while others may be less common with small magnitude. 
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LES Training Process 

The theory of change depicted by this logic model begins with the LES training process, which has been 
shown to affect training effectiveness. This component of the model refers to the characteristics of the 
training activity, the resources employed to provide the training, and the participants' engagement in 
and reaction to the training activity. According to the literature, the following features are important 
characteristics of LES training activity: 

■ Skills being taught: As some Essential Skills are likely to be more discrete and can be taught more 
readily than others and can therefore be more readily transferred, the type of skill being taught may 
affect the magnitude of the skills gains resulting from the training. 

■ Business alignment: The extent to which the learning objectives of the training program support 
business and performance needs of the organization, including improved customer service and 
customer/client relations and increased productivity. Employers were asked to articulate in 
behavioural terms what workers should do in order to contribute to stated business needs. 

■ Match to learner needs and goals: Training should only be delivered to workers with 
performance gaps, meeting their skills needs and objectives. 

■ Program design and delivery: Once the training needs are determined, the goals of instruction 
should be identified and tied to desired performance goals, making explicit how training will close 
the worker performance gap. Then the training is designed, structured, and delivered to the right 
audience, at the right time. 

■ Training duration and intensity: Duration refers to the amount time spent in the training activity 
and intensity refers to the amount of training in a particular amount of time and can be measured in 
terms of hours per week/month. 

■ Instructor: If an instructor engages with staff and gets to know workplace processes, he or she can 
better ensure a fit between the teaching content, the learners' needs and the company's aims. 
Moreover, the instructor can provide feedback on the learners' progress in the training course to 
supervisors on the one hand and gain feedback on job demands/issues on the other. 

Program Outcomes 

Program outcomes of the LES training are relevant skills levels, behaviours, and/or characteristics 
measured following a training activity, such as a literacy score, self-confidence, earnings, and 
participation in workplace/everyday activities. Program outcomes may occur for individuals, groups, 
families, households, organizations, businesses, or communities as a result of a training program. 
Outcomes may be intended or unintended, positive or negative. In the Measures of Success project, we 
focused primarily on outcomes that occur for individual learners and participating businesses, inside 
and outside the workplace. Outcomes can be further divided into two intermediate and long-term, as 
well as mediating/moderating factors which can play a role in how training leads to the outcomes of 
interest. A key point is that the training intervention is only one part of a larger system leading to 
expected outcomes where other factors play a role in influencing worker behaviour, worker 
performance, and business outcomes. This is an important consideration when measuring the benefits 
to training. 
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Figure 1 A logic model for estimating outcomes and returns to LES training in the workplace 


1. Workplace LES training process 



3. Workplace factors 

1 

• 

Employer/manager 

expectations 

i 

• 

Coaching and 
reinforcement 


• 

Workplace culture 


• 

Resources and 

i 


supports 

j 


4. Enabling and hindering 
factors 



6 






Measures of Success 
Final Report - March 2013 


a) Intermediate Outcomes 

Intermediate outcomes are the changes in the level of relevant skills, behaviours, and/or characteristics 
that not only have value in their own right, but they may also support the attainment of the long-term 
outcomes of interest. Chronologically, we might expect these outcomes to occur either during, 
immediately after or shortly after the program. Workplace training has been hypothesized to lead to 
various intermediate outcomes including those related to human capital (increased knowledge and 
skill level), social capital [increased network size and improvement in network quality), and 
psychological capital [such as changes in self-esteem and self-confidence). The logic model also 
includes two sets of overlapping intermediate outcomes: workplace performance and practices that 
individuals engage in their everyday lives. The two sets of outcomes are depicted as overlapping to 
illustrate an ambiguous delineation between learners' personal and workplace practices and 
behaviours. These may include a range of behaviours that provide further opportunities for the 
practice/use of literacy skills that may support learning. 

b) Mediating/Moderating and Contextual Factors 

The relationship between workplace literacy training and longer term outcomes may be influenced by a 
host of mediating and moderating factors, at both the individual and workplace levels, which are those 
factors that can affect and be affected by the training. Many of these are part of the training's context or 
are intermediate outcomes. 

Mediating factors explain how or why a relationship may exist between explanatory or predictor 
variables and the variable of interest [the dependent variable). These include individual factors such as 
learners' engagement with the learning activity [e.g., attendance, active participation, completion of 
learning tasks). Workplace mediating factors, which explain how workers apply what they have gained 
from training to the job, include management expectations [e.g., awareness, intentionality, 
engagement) related to the training intervention, as well as clarity of roles and expectations of staff; 
performance and training incentives; work systems and processes; workers' access to information, 
people, tools and job aids; and coaching and reinforcement. According to the literature, the absence of 
performance support is often the greatest block to exemplary work performance. Many of the 
intermediate outcomes mentioned above, such as social capital, psychological capital, and various 
literacy practices, are also in fact mediating factors for other longer-term outcomes, as they may 
interact and further reinforce learning. 

Moderating factors specify the conditions under which an explanatory variable influences a dependent 
variable. Moderating factors are those that may influence the relationship between intermediate and 
long-term outcomes for individual learners and their families. Individual characteristics such as gender, 
age, initial skill and educational level, and attitude toward learning, as well as individual lifecycle 
circumstances relating to household income, marital status and family status, are examples of 
moderating factors because they can affect whether the training results in positive or negative, weak or 
strong outcomes. Moderating factors also include the socioeconomic context; the policy, program and 
institutional environment; and workplace factors such as employment size. For business outcomes, 
external factors like market conditions as well as the social, political, policy, and institutional context 
may moderate the relationship between changes in workplace performance and longer term outcomes 
of training for businesses. 
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Moderating factors are typically not affected by training but can influence its effectiveness. A moderator 
may reduce or enhance the direction of the relationship between a predictor variable and a dependent 
variable, or may change the direction of the relationship between the two variables from positive to 
negative or vice versa. 

Longer-Term Outcomes 

Longer-term outcomes are those that may take longer to occur than intermediate outcomes, but are the 
outcomes that individuals, businesses and society ultimately care about. In the case of workplace 
training, long-term outcomes can accrue to both employers (businesses) and individual learners, and 
the outcomes may be financial and non-financial, and more or less tangible. Individual and business 
outcomes may also be interrelated and in some cases mutually reinforcing. 

■ Non-financial individual outcomes are defined here as the outcomes that are experienced by an 
individual worker or their family that do not directly affect one's wealth or income, such as 
improved individual or family health, and improved relations with family, friends, and colleagues. 
Also, the broader adult learning literature identifies several non-market outcomes of adult learning 
programs, including increased access to services, increased life satisfaction, improved health, lower 
stress, and improved relationships with family, friends and coworkers. 

■ Financial individual outcomes refer to long-term outcomes that affect an individual's wealth or 
income. Potential financial benefits for workers reported in the literature include better job quality 
(such as a safer workplace], career advancement and higher wages. 

■ Tangible business outcomes include increased productivity, increased sales, cost control, 
improved product quality, improved customer service, worker retention, reduced absenteeism, and 
improved health and safety. 

■ Less tangible business outcomes include outcomes that cannot be easily quantified or monetized 
such as improved workplace morale, cohesion among co-workers, improved relations and trust 
between management and employees, and an enhanced culture of learning. 

It should be noted that the current study measured "long-term" outcomes at only 6-months following 
the onset of training due to the constraints of the overall project timelines i.e. a 3-year study period that 
included development of a framework and instruments. Given this context, it was a significant 
challenge to detect long-term changes, particularly given the fairly modest number of training hours 
provided (i.e., under 40 hours per participant) compared to post-secondary forms of skills upgrading 
(e.g., vocational, college diplomas). 

c) Return on Investment/Expectations 

The return on investment (ROI) of training refers to the net cost or benefit of the training activity 
relative to the cost, and is frequently expressed as a ratio or a percentage. All benefits of the training 
are given a monetary value, summed, and compared to the costs, including the actual expenditure on 
(investment in) the training, to determine whether the program yielded a net benefit or net cost. 

It is important to note that most research on the returns to workplace literacy programs use a 
qualitative methodology that draws on employers' perceptions. This is in part because few companies 
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collect quantitative data on the benefits arising from the training that they deliver to employees, and 
also because estimating ROI tends to be complex. Also, due to the difficulty of converting intangible 
benefits to monetary values, they are often excluded from ROI calculations, likely leading to 
underestimation. Another fundamental issue is that it can be difficult to establish causality between 
outcomes and the training intervention without an experimental design, which was outside the scope of 
the current project. 

Return on expectations (ROE] is the process of estimating returns to training relative to stakeholder 
expectations. Unlike ROI which is simply an accounting valuation technique, ROE is an evaluation 
process that ideally begins before the training intervention is implemented, since it requires the 
training program to be tied to performance and business needs as expressed by key stakeholders. The 
term ROE was created to highlight the importance of aligning training goals and content to the specific 
needs of the organization by ensuring that the training aims to address the causes of performance gaps 
and in turn contribute to business goals, which are what employers care about. The Measures of Success 
project is in large part an exercise in evaluating ROE. 

2.2 Expectations - the Challenge in Quantifying What Matters and How Much 

Beyond developing this theory-driven research framework, the key challenge in applying it, 
analytically, is in determining when expectations have been met - not only on the longer-term 
outcomes of interest to the stakeholder, but on those intermediate outcomes that make up the change 
process. For instance, regarding implementation factors, how effectively does alignment between 
training and business results need to be? Or relating to mediating variables, "how much” social capital 
or literacy practices make a difference in supporting other positive outcomes? And with respect to 
longer-term outcomes, how much performance improvement for the business or increased well being 
for individuals is considered a success? 

These "thresholds" for successful change on intermediate and longer-term outcomes can come from 
several sources including stakeholder expectations, established benchmarks, or statistical comparisons 
with outcomes of related programs and participants. Of these three, stakeholder expectations are the 
foundation of analyzing change with this method. For Measures of Success, we collected expectations 
from several stakeholders including the Steering Committee, provincial coordinators, workplace 
educators, and employers. Expectations of employers - what motivates them and what they hope to 
achieve with the training - were paramount. 

However, while key stakeholders can provide clear direction in determining what matters, it is 
considerably more difficult for them to express how much something matters. As a result, stakeholder 
expectations are often used as in conjunction with other benchmarks to determine if the magnitude of 
change is relevant. It is most common in theory-driven case studies to compare outcomes to various 
benchmarks, either national averages, or, particularly for this project, to average outcomes for related 
programs and businesses. 

In summary, benchmarking occurs at three levels: (1) the foundation for constructing a broad 
framework comes from prior research and expert consultations (steering committee, provincial 
practitioners); (2) the guidance for determining what matters most comes from site-level stakeholders 
(employers); and (3) the relevance of change can be informed by benchmarking against related 
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programs and businesses - namely, in this project, the results from the Essential Skills training 
initiatives in other participating businesses in Nova Scotia and Manitoba. 

2.3 Research Data Collection 

To address the questions raised above, researchers collected data from a variety of sources using 
multiple techniques. The primary vehicle was structured and semi-structured interviews to collect both 
qualitative and quantitative data. Secondary data sources supplement the interview evidence wherever 
possible. Interviews were conducted at baseline, three months, and six months after training onset. 
Table 2.1 lays out the various data sources and the issues addressed in each source. 

T able 2.1 Data Sources and Issues Addressed 


Data Collection Vehicle and Source, 
per Site 

Issues Addressed 

Project team meeting/interview {only at 
baseline) 

Coordination 

Semi-structured interview with project 
co-coordinator {only at baseline) 

Purpose, design and delivery of the local training project. 

Semi-structured interview with 
instructor/trainer 

All issues around the delivery of the training including the instructors' teaching and 
delivery style, curriculum and customization, as well as delivery successes, 
challenges and issues. 

Semi-structured interview project 
sponsor (senior employer contact: 
CEO/owner) 

Business goals, performance needs and performance gaps and how these may 
relate to the training. 

Semi-structured interview with line 
supervisor(s) 

Issues related to performance needs similar to interviews with senior contact 
interviews but at a more “micro” level. 

Organizational profile worksheet 
completed by senior business 
representative {only at baseline) 

Business characteristics such as size, workforce traits and training provided. 

Survey (structured interview with “pencil 
and paper” completion) with workers in 
groups 

Issues related to workers' outcomes, including sociodemographic and employment 
characteristics; expectations for the training (at baseline); human, psychological and 
social capital; views of the training (post-training); etc. 

Administrative data 

Participant's attendance and the number of hours of training received (intensity, 
duration). 

Already completed ONA protocol with 
representatives of businesses 

Workplace background, local project design as well as workplace issues. 

Other secondary data sources 

Local socio-economic conditions as well as the broader market conditions for the 
relevant industries, in order to provide context. 
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Chapter 3: Project Implementation and Training Profile 

This chapter describes the implementation processes and activities that led up to the delivery of the 

training programs that were used for the Measures of Success project, from the recruitment of 

businesses and participants, to the needs analysis, and ultimately the training plans in terms of what 

and how it was delivered. It addresses the following questions: 

■ How were employers and employees recruited for the training? 

■ What were employers' training motivations and business and performance needs? 

■ Which Essential Skills does the training target? 

■ What are some of the characteristics of the training, such as the length, location, and schedule? 

3.0 Highlights 

■ Number of businesses: A total of 18 employers were recruited for the project - 8 from Nova Scotia 
and 10 from Manitoba. Most sites selected in both provinces already had experience with 
workplace Essential Skills training. In at least 10 of the sites, employers had previously participated 
in training programs delivered by either Manitoba or Nova Scotia. 

■ Number of trainees: The number of training participants ranged from 6 to 20 in Nova Scotia 
businesses and 7 to 30 in Manitoba. On average, about 12 participants were recruited per business, 
with two businesses with more than 20 participating employees. Participation was reported to be 
voluntary for the majority (14 of 18) businesses, although in many cases, employees were 
encouraged to participate by their managers. Sometimes this encouragement was targeted to 
specific groups of employees. 

■ Employer motivation for participating: All businesses were motivated to participate in the 
training to help address a pressing business need (e.g., to be more competitive, implement new 
systems in response to rapid growth). In most cases, the motivation was related to a specific 
business need [e.g., preparing employees to use new computerized technology). In other cases, a 
more general motivation for training was reported [e.g. improving communications both internally 
and with customers). Most employers were responding to an existing need, although 4 employers 
were preparing for a future need. 

■ Essential Skills targeted: A wide range of Essential Skills was targeted by the training. The most 
common were oral communication skills, followed by thinking skills [including problem-solving 
and critical thinking), digital technology skills, and working with others. 

■ Training hours, class sizes: Instructors and coordinators in Nova Scotia reported average total 
training hours per business [39) that was about three times greater than reported in Manitoba [11). 
Class sizes ranged widely, from as few as five participants to as many as 26 per class, with an 
average of 12 to 13 participants per class. 
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3.1 Site Selection 

SRDC in partnership with the Project Steering Committee identified the following site selection criteria: 

■ Size: In order to allow for a sufficient sample, SRDC suggested preference be given to medium-sized 
and larger workplaces and ones where there would be at least 10 employees participating in 
training. 

■ Willingness and Stability: Each worksite had to be willing to meet the research requirements. As 
well, preference was given to workplaces that were relatively stable and likely to remain in 
operation for the duration of the study. 

■ Stratification: Selection of businesses for this project was stratified based on industry and whether 
the worksite is a new or existing participant in LES training. In consultation with project partners 
the following stratifications were agreed upon: industry. Essential Skills targeted, and history with 
the respective provincial training department. 

The original criteria and numbers initially recruited are presented in Table 3.1 below. Characteristics of 
the training are described later in this chapter, while the characteristics of participating businesses are 
described in Chapter 4. 

In addition, SRDC ensured that, wherever possible, workplaces were selected purposefully to ensure a 
diverse cross-section of workplaces. Programs differed on a range of factors of which researchers were 
mindful in site selection, including the following: 

■ Program characteristics: program length, program intensity, and the type of employer 
participation (i.e., whether it was a single employer or a consortium); 

■ Business characteristics: unionization, geography, and profitability; and 


■ Worker characteristics: immigrant and Aboriginal status, and gender composition. 

T able 3.1 Original Stratification Criteria, for each province 


Industry, Positions 

Essential Skills Targeted 

Training History with Department 

Health and Social Services, 
financial and administrative 
positions 

■ Document use 

■ Oral communication/ working with others 

■ Digital Technology 

■ (Numeracy) 

■ 2 firms new to LES training 

■ 1 firm with prior experience 

Manufacturing, Natural Resources 

■ Document use 

• Oral communication/ working with others 

■ Digital Technology 

■ (Numeracy) 

■ 3 firms new to LES training 

■ 1 firm with prior experience 

Service and Sales 

■ Document use 

■ Oral communication/ working with others 

■ Digital Technology 

■ (Numeracy) 

■ 2 firms new to LES training 

■ 1 firm with prior experience 
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3.2 Employer and Employee Recruitment 

Once the Steering Committee determined that a potential worksite met the selection criteria, the 
provincial project team then approached the employer and requested their participation in the 
Measures of Success project. Each province aimed to recruit 10 workplaces into the study. Over the 
period from January to December 2011 period, the project team recruited a total of 18 employers - 
8 from Nova Scotia and 10 from Manitoba.i In at least 10 of the sites, employers had previously 
participated in a training program delivered by either Manitoba or Nova Scotia. 

The number of employees recruited for training from Nova Scotia businesses ranged from 6-20 and in 
Manitoba, from 7-30. On average, roughly 12 participants were recruited per business, with two 
businesses having more than 20 employees participating in the training. Specific criteria for participant 
selection were not provided or imposed. In most businesses, participation in the training was 
voluntary. However, several instructors indicated that even when participation was voluntary, 
employees were encouraged to participate by their managers. In some cases, specific groups of 
employees were encouraged versus other groups. The selection of participants was ultimately 
determined by the employer, often with some facilitation from program coordinators, in completing 
organizational and individual needs assessments. 

As indicated in Table 3.2, in two-thirds of cases, instructors and coordinators reported that they 
perceived employees to be well informed of the training opportunity and its objectives. In the 
remaining third, employees at four businesses (MBl, M4, MB8 and MB13) were not perceived to be 
well-informed while in two businesses [MB3 and MBS), instructors and coordinators were uncertain. 


The following chapter provides a complete profile of the participating businesses and workers. 

Table 3.2 Employee Recruitment 


Business 

Code 

Number 

Recruited* 

Employees Weii 
informed? 

Seiection/Sign-up 

NS1 

12 

Yes 

■ 

Voiuntary 

NS2 

8 

Yes 

■ 

Voiuntary, but with encouragement from management 

NS3 

12 

Yes 

■ 

Voiuntary 

NS4 

6 

Yes 

■ 

Voiuntary, but with encouragement from management 

NS5 

9 

Yes 

■ 

Voiuntary 

NS6 

20 

Yes 

■ 

■ 

Mandatory for supervisors 
Voiuntary for frontiine staff 

NS7 

12 

Yes 

■ 

Voiuntary 

NS8 

20 

Yes 

■ 

Voiuntary 


' The baseline report, SRDC (2012), provides a short profile of each of the participating businesses. 
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Business 

Code 

Number 

Recruited* 

Employees Weii 
informed? 

Seiection/Sign-up 

MB1 

7 

Yes, but not in aii cases 

- Mandatory 

MB2 

10 

Yes 

■ Voiuntary, but with encouragement from management 

MBS 

8 

Uncertain 

■ Voiuntary, but with encouragement from management 

MB4 

24 

Yes, but not in aii cases 

■ Voiuntary, but with encouragement from management 

MBS 

29-30 

Uncertain 

■ Voiuntary 

MBS 

8-12 

Not generaiiy 

■ Voiuntary, but with encouragement from management 

MB11 

9-10 

Yes 

’ Mandatory 

MB13 

8 

Yes, but not in aii cases 

■ Voiuntary 

MB14 

18 

Yes 

■ Voiuntary, but with encouragement from management 

MB15 

16-18 

Yes 

■ Mandatory 


Source: Administrative data and MoS Instructor and Provincial Coordinator Protocols. 


3.3 Motivations for Training and Business Aiignment 

The ultimate goal is not simply to get employees to attend a training program but to enable them to 
strengthen skills to perform better on the job and to benefit the business. To achieve this goal, both 
provinces started by gaining an understanding of the business needs of each participating company and 
exploring how Essential Skills training of their employees might be part of the solution to meet these 
needs. This approach is consistent with the training and development literature that suggests that the 
more training interventions are aligned with business needs, the more likely it is to generate positive 
outcomes for the business. Indeed, a recent study by the Canadian Society for Training and 
Development concludes [2010), "to deliver meaningful value, learning investments must evolve directly 
from the organization's key business priorities." 

Consistent with the literature, we identify three components to business alignment: 

■ Business need - To what extent is the motivation for training related to a pressing business need 
versus a 'nice-to-have' or general interest? 

■ Performance-skills link - Is there a clear understanding of what employees need to be doing 
differently on the job to address business needs? And is there a clear understanding of how 
improving Essential Skills can improve this on-the-job performance? Or, alternatively, is there 
simply a vague hope that training can help? 
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■ Curriculum customization - Is the curriculum customized to develop Essential Skills specifically 
needed to support clearly identified on-the job performance areas? Or, alternatively, is the training 
more generic and less connected to on-the-job performance? 

Once an employer agreed to participate, coordinators in Manitoba and Nova Scotia visited the worksite 
to identify business and performance needs through a formal needs analysis. Through this process, 
consultants identified the employer's key needs and determined whether and how ES training could 
help to address them. Findings were used to customize the training to meet specific performance and 
skills needs. If properly executed, this process should promote business alignment. 

Is motivation for training related to a pressing business need? 

The first component of business alignment is the extent to which the motivation for the training was 
related to an important business need versus a "nice-to-have" outcome but not necessarily critical to 
business operations. Business needs are the business goals and objectives that the organization 
believes it must achieve if it is to be successful. A key finding is that, in all 18 cases, project sponsors 
stated that they were motivated to participate in the training to help address a pressing business need.2 
However, sponsors differed in the extent to which the needs were in response to very specific versus 
more general needs. In the majority of businesses (12), the motivation was related to a very specific 
business need, such as preparing employees to use new computerized technology or reducing errors 
with blueprints. In the remaining businesses (6), project sponsors reported a more general motivation 
for training such as improving communications. Most employers were responding to an existing need, 
although 4 employers were preparing for a future need. 

Is there a link between business needs, employee performance and Essential Skills? 

The second part of business alignment relates to performance needs. Performance needs are the on- 
the-job accomplishments required of individuals who are performing a specific job and who contribute 
to the achievement of the business goals. Performance needs identify what individuals must do more, 
better, or differently if the business goals are to be achieved. Essential Skills training aims to provide 
the foundational skills that employees need to improve specific on-the job performance. Thus 
instructors should understand what specific performance needs of the organization and its employees 
are to ensure the training targets skills development that will support these specific performance areas. 

For the majority of businesses (14), project sponsors clearly described a need in at least one specific 
on-the-job performance area. An additional three project sponsors identified more general 
performance issues. Only one project sponsor did not identify any performance issues. From the 
perspective of instructors, ten were able to clearly describe specific performance needs in terms of 
what employees need to be doing more or less of, or differently. An additional four instructors 


The analyses in subsequent sections of the report focus on the extent of alignment between business 
needs and the training provided, as measured by consistency in responses of project sponsors and 
supervisors in these three areas. For a complete description of the business needs of each participating 
site see SRDC (2012). 
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described performance needs in general terms only. Three instructors did not describe any on-the-job 
performance need at all. 

Is the curriculum customized to both job performance and skills needs? 

The third part of the business alignment equation relates to the curriculum. Was the curriculum 
customized based on an understanding of the business needs that motivated the training and is it 
designed to develop Essential Skills specifically needed to support clearly identified on-the job 
performance areas? Or is the training more generic and less connected to on-the-job performance? 

Just over half of the instructors (in 10 businesses) clearly articulated the business need that motivated 
the training for employers. Two-thirds of instructors indicated that the curriculum would be 
customized to reflect specific on-the-job performance needs. These instructors indicated that training 
customization was the based on a comprehensive needs analysis [e.g., consulting widely with 
managers, analyzing ONA results) and/or that the curriculum would be focused on working with a 
specific workplace material related to the performance need (e.g., working with blueprints). Two-thirds 
of instructors (12) indicated that they would integrate actual workplace materials and/or scenarios 
into the training (e.g., using tape measures, working with reports), which is a way to customize the 
training to the workplace. All but three instructors said that they would tailor the training to meet 
employees' skills levels (e.g., using results of Individual Needs Assessments to inform the training level 
of complexity). 

Chapter 7 will explore the relationship between the extent of business alignment and post-training 
outcomes. Table 3.3 presents a summary of each of these elements of alignment for all participating 
businesses. 

3.4 Description of Training 

Table 3.4 presents information on the training delivery at each business as reported in administrative 
records and by instructors and coordinators. The table shows that a wide range of Essential Skills were 
targeted by the training. The most common were oral communication skills, with more than half of all 
training (10 businesses) targeting these. Also common were thinking skills (including problem-solving 
and critical thinking), digital technology skills, and skills for working with others, with at least one- 
third of training programs targeting these(7, 6 and 6 training programs, respectively). 

In Nova Scotia, on average the total number of training hours is much greater than that in Manitoba. 
Instructors and coordinators in Nova Scotia reported total training hours about three times greater 
than those reported in Manitoba (an average of roughly 39 hours per business in Nova Scotia versus an 
average of roughly 13 hours in Manitoba). Individual training sessions are also generally longer in Nova 
Scotia than in Manitoba, with training sessions averaging roughly 4 hours each in Nova Scotia and 
roughly 2 hours each in Manitoba. The duration of the training, that is, the period of time over which 
the training was delivered, was also generally longer on Nova Scotia (an average of roughly 10 weeks in 
Nova Scotia versus roughly 7 weeks in Manitoba). 
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Table 3.3 Summarizing Training-Business Alignment: Business Needs, Performance-Skills Link, and Intended Curriculum Customization 


Code 

Rating of Degree of 
Training-Business 
Aiignment (preliminary) 

Sponsor clearly articulates 
a need in at least one 
specific on-the-job 
performance area? 

Instructor describes 
performance in terms of 
what employees should be 
doing more/less of or 
differently? 

Instructor 
described 
business need 
that motivated 
training? 

Does the instructor indicate curriculum is customized 
to reflect ... 

Specific on-the-job 

performance 

needs? 

Workplace 
materials and 
scenarios? 

Employee 

skill 

levels? 

NS1 

Well 


Yes 

Yes 

Yes 

Yes 

Yes 

NS2 

May be at risk 

Yes 

Somewhat 

Yes 

No 

No 

Yes 

NS3 

May be at risk 

Yes 

Somewhat 

Yes 

No 

No 

No 

NS4 

Well 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

NS5 

Well 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

NS6 

Moderate 

Somewhat 

Somewhat 

Yes 

Yes 

Yes 

No 

NS7 

Moderate 

Yes 

Somewhat 

Yes 

Noi 

Yes 

Yes 

NS8 

Moderate 

Yes 

No 

Yes 

No2 

No 

Yes 

MB1 

Well 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

MB2 

Moderate 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

MBS 

May be at risk 

No 

No 

Yes 

No 

Yes 

No 

MB4 

Moderate 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

MBS 

Moderate 

Somewhat 

Somewhat 

Yes 

Yes 

Yes 

Yes 

MBS 

Moderate 

Yes 

Yes 

Yes 

Yes 

No 

Yes 

MB11 

Well 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

MB13 

May be at risk 

Somewhat 

No 

Yes 

No 

Yes 

Yes 

MB14 

Well 

Yes 

Yes 

Yes 

Yes 

Yes 

Yes 

MB15 

Moderate 

Yes 

Yes 

Yes 

Yes 

No 

Yes 


' The training is not designed to support specific on-the-job needs because the training is in anticipation of a future need to access online training and this training has not been designed yet. Instead 
employees will build basic foundational skills that they will likely need in the future. 

2 The training is not designed to address specific on-the job needs because the training is motivated by a need to develop basic skills to support implementation of computerized technology in the future. 
Source: MoS ONA Protocols, Project Sponsors and Instructors (n=18). 
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Training was delivered largely in group sessions though some 1-on-l was provided based on individual 
needs. Class sizes ranged widely, from as few as five participants to as many as 26 per class, with an 
average of 12 to 13 per class.^ 


T able 3.4 Profile of T raining Delivery 


Code 

Essential Skills Targeted 

T ralnees* 

Location 

Hours 

Training intensity/duration 

NS1 

■ 

Digital technology 

17 

Onsite 

30 

one 3-hrfor10wks 

NS2 

■ 

Oral communication 

9 

Offsite 

42 

two 3-hr sessions for 7 wks 

NS3 

■ 

■ 

Oral communication 
Document use 

12 

Offsite 

40 

one 8-hr session for 5 wks 

NS4 

■ 

Numeracy 

10 

Onsite 

40 

two 2-hr sessions for 10 wks 

NS5 

■ 

■ 

Digital technology 
Thinking skills 

8 

Onsite 

40 

one 4-hr session for 10 wks 

NS6 

■ 

Oral communication 

22 

Onsite 

40 

one 4-hr session for 10 wks 

NS7 

■ 

Digital technology 

12 

Onsite 

39 

one 3-hr session for 13 wks 

NS8 

■ 

Digital technology 

26 

Onsite 

40 

one 3-hr session for 13 wks 

MB1 

■ 

■ 

Writing 

Thinking skills 

5 

Onsite/ 

offsite 

16 

two 2-hr sessions for 4 wks 

MB2 

■ 

■ 

■ 

Thinking skills 
Oral communication 
Working with others 

10 

Offsite 

(Head 

office) 

4.5 

one 1 .5-hr sessions for 3 wks 

MBS 

■ 

■ 

■ 

■ 

Oral communication 
Digital technology 
Document use 
Working with others 

8 

Onsite 

18 

one 2-hr session for 9 wks 

MB4 

■ 

■ 

■ 

Numeracy 
Reading, writing 
Critical thinking 

9 

Onsite 

16-20 

3-4 hr sessions for 5 wks 

MBS 

■ 

■ 

■ 

■ 

■ 

Reading 
Document use 
Numeracy, 

Oral communication 
Working with others 

26 

Offsite 

15 

one 1 .5-hr session for 10 wks 


3 Note that in fact these are the counts of those participating in the baseline survey, which is fairly close to 
the actual number of trainees. 
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Code 

Essential Skills Targeted 

Trainees* 

Location 

Hours 

Training intensity/duration 

MBS 

■ Digital technology 

10 

Onsite 

3 

one 3-hr session 

MB11 

■ Communication 

■ Working with others 

■ Problem solving 

■ Conflict resolution 

9 

Offsite 

(another 

branch) 

6 

two 3-hr sessions 

MB13 

■ Oral communication 

■ Document use 

■ Reading 

■ Working with others 

9 

Onsite 

8 

one 1-hr session for 8 wks 

MB14 

■ Document use 

■ Numeracy 

■ Oral communication 

16 

Onsite 

20 

one 2-hr session for 10 wks 

MB15 

■ Working with others 

■ Oral communication 

■ Thinking skills 

17 

Onsite 

20 

one 2.5-hr session for 8 wks (over a 4- 
month period) 


* The number of participants who responded to the baseline survey. This differed by up to 6 from the number originally indicated in the instructors 
and provincial coordinators baseline protocols and as well as from the number who actually participated in the training in two worksites. 
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Chapter 4; Sample Characteristics and Contextual Factors 

This chapter describes the businesses and workers participating in the Measures of Success project 
including the industry, structure and size of the firms as well as some key socio-demographic 
characteristics of the learners. This information provides not only an overall picture of the sample but 
also a look at some of the key contextual factors that may influence the pattern of training outcomes, 
whose relationships will be analyzed in subsequent chapters. 

4.0 Highlights 

■ Of the 18 businesses participating in the project, just over half were in manufacturing with 
the remainder in services. The size of participating businesses varied greatly from a low of 
43 employees to a high of 1,421. Based on Industry Canada criteria, half the businesses would be 
considered small, with fewer than 100 employees; five were medium-sized with more than 100 but 
less than 500 employees, and four were large with greater than 500 employees. 

■ The proportion of the total workforce enrolled in each business varied greatly and was quite 
small in some sites, which reduces the ability to detect business-level effects at these sites. 

On average, participants represent 4.2 percent of a business's overall workforce, with the 
proportion being twice as high in Nova Scotia than in Manitoba. The proportion participating 
ranged from less than 1 percent in three large Manitoba-based businesses to over 30 percent at one 
site in Nova Scotia. 

■ Participants were quite similar to the broader workforce in participating businesses, though 
there was a greater proportion of women, they were somewhat younger, and fewer had 
English as a Second Language (ESL). Comparing the Measures of Success sample to the broader 
workforce reveals a lower proportion of male participants and fewer over the age of 40. Also, the 
proportion of participants who speak English as their second language at home [ESL) was lower. 
Businesses differ across provinces in this respect: all ESL participants were in Manitoba, a challenge 
for ES training and research that was not present in the Nova Scotia sample. 

■ There was wide variation in education levels and wages - the proportion with a non- 
university credential was higher than the Canadian average, while the proportion with a 
university degree and the average wages were lower. While a larger percentage of participants 
had no education credential or only a high school diploma compared with working-aged Canadians 
overall, a higher percentage had some form of trade or vocational certificate. Manitoba participants 
had higher educational credentials. Similarly, the average hourly salary of participants was $18.34 
which is lower than the Canadian average of $22.72, while wages were higher in Manitoba. 

■ Participants were in fairly stable jobs - over 90 percent in permanent positions and 

96 percent employed full-time, which supports persistence in training and the research 
follow-up. The average tenure was almost 10 years and half the participants had been working for 
the business for at least five years. While this stability bodes well for worker retention and 
observing longer-term outcomes, tenure was a good deal lower among participants in specific 
workplaces. 
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■ Participants were employed in a wide range of positions, which suggests potential wider 
applicability of the results. The positions were clearly aligned with the sector of their employer. 
There were case workers and counsellors from the two social services businesses and labourers, 
pipefitters, various production workers, lead hands and engineers from the other businesses, which 
were all in manufacturing. 

4.1 Business Sector, Size, and Growth 

Table 4.1 presents the sector and size of each participating business, as well as whether the employer 
reported that the business experienced a recent expansion or contraction. As the table shows, 
businesses ranged widely in size from as few as 43 employees to over 1,400 employees. The overall 
average business size was 296 employees, and the overall median was 103 employees. Based on 
Industry Canada criteria, half the businesses would be considered small, with fewer than 
100 employees; five would medium-sized, with more than 100 but less than 500 employees; and four 
would large businesses with greater than 500 employees. On average, businesses in Manitoba 
(374 employees) are much larger than businesses in Nova Scotia (199 employees). Roughly half the 
businesses in each province are in the manufacturing sector and half in the services sector. 

Overall, half the businesses were reported to have experienced recent expansion. Expansion was 
reported slightly more often in Manitoba, with more than half of Manitoba businesses reporting 
expansion and just under half reporting this in Nova Scotia. Only two employers (NSl and NS2) 
reported contraction, both of which are in Nova Scotia. The remaining businesses reported neither 
expansion nor contraction, while one business (MB2) reported experiencing both. 


T able 4.1 Sector, Size, and Business Growth 


Code 

Sector 

No. of Employees 

Growth 

NSl 

Manufacturing 

226 

Contracted 

NS2 

Manufacturing 

43 

Contracted 

NS3 

Services 

55 

Expanded 

NS4 

Manufacturing 

909 

Expanded 

NS5 

Services 

137 

No change 

NS6 

Services 

70 

No change 

NS7 

Manufacturing 

58 

Expanded 

NS8 

Services 

95 

No change 

Nova Scotia mean 

199 
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Code 

Sector 

No. of Employees 

Growth 

MB1 

Manufacturing 

700 

No change 

MB2 

Services 

279 

Both 

MBS 

Manufacturing 

1,421 

... 

MB4 

Manufacturing 

63 

Expanded 

MBS 

Manufacturing 

111 

Expanded 

MBS 

Manufacturing 

46 

No change 

MB11 

Services 

65 

Expanded 

MB13 

Services 

750 

Expanded 

MB14 

Manufacturing 

225 

Expanded 

MB15 

Services 

75 

Expanded 

Manitoba mean 


374 


Overall mean 


296 



Source: Measures of Success Organizational Profile Worksheet and administrative records. 


4.2 Internal and External Changes 

Participating businesses reported a wide range of recent internal and external changes related to the 
business. Internal changes, such as automation of processes, occur usually within the direct influence of 
the employer. External changes, such as market or regulatory changes, occur generally outside the 
employer's direct influence. Internal and external changes may be related in some cases. For instance, 
in several businesses an external change, such as increased market competition, may motivate an 
internal change, such as adoption of automated processes to improve competitiveness. 

Employers reported facing a variety of organizational (internal) and sectoral (external) changes. 
Common organizational and sectoral changes that businesses identified include the following: 

■ Rapid growth arising from economic recovery (most of the manufacturing business reported this 
condition); 

■ Implementing new technologies and computerization; 

■ Increased competition and globalization; and 

■ New security requirements and increased regulatory requirements. 
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4.3 Participant Demographic Characteristics and Workforce Comparisons 

Table 4.2 indicates that the 226 participants in the Measures of Success project represent a little over 
four percent of the combined workforces of their businesses' numbered at 5,328. On average, there 
were 12.5 participants per business, but the number participating ranged widely. In seven businesses 
(NS2, NS4, NS5, MBl, MB2, MBS, MB13 and MB14), participants represented no more than about seven 
percent of their business's workforce. Thus, likelihood of training having an observable change at the 
workplace level in these businesses will be limited and difficult to detect. This argues for added 
emphasis, in the site-level analysis, on measures at the participant level, e.g., individual performance, 
which is correlated with business-level results. In contrast, about 20 percent or more of staff in eight 
businesses (NSl, NS4, NS5, MBl, MB2, MB3, MB13 and MB14) participated in training. Aggregate 
measures of outcomes in these businesses may be more sensitive to change produced by the training 
intervention. 

Comparing provinces, participants represent twice as large a proportion of the workforce of Nova 
Scotia businesses as in Manitoba businesses, suggesting the training has a greater chance of effecting 
change at the workplace level in the former province. Note that, while the average number of 
participants is similar in the two provinces [about 13 and 12], the participant share is greater in Nova 
Scotia because the average workforce of participating businesses is considerably larger in Manitoba 
[374 vs. 199 employees). 


Table 4.2 Percentage of Employees Participating in MoS Training at each Site 


Province and Business 

No. of Participants* 

No. of Employees in Workforce 

% of Workforce 

Nova Scotia 

NSl 

14 

226 

6.2 

NS2 

8 

43 

18.6 

NS3 

12 

55 

21.8 

NS4 

10 

909 

1.1 

NS5 

9 

137 

6.6 

NS6 

22 

70 

31.4 

NS7 

12 

58 

20.7 

NS8 

20 

95 

21.1 

NS Totais 

107 

(avg = 13.4) 

1,593 

(avg = 199.1) 

6.7 

(avg = 15.9) 
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Province and Business 

No. of Participants* 

No. of Employees in Workforce 

% of Workforce 

Manitoba 

MB1 

5 

700 

0.7 

MB2 

10 

279 

3.6 

MBS 

8 

1,421 

0.6 

MB4 

9 

63 

14.3 

MBS 

26 

111 

23.4 

MBS 

10 

46 

21.7 

MB11 

9 

65 

13.8 

MB13 

9 

750 

1.1 

MB14 

16 

225 

7.1 

MB15 

17 

75 

22.7 

MB Totais 

119 

(avg = 11.9) 

3,736 

(avg = 373.5) 

3.2 

(avg = 10.9) 

Totais 

226 

(avg = 12.6) 

5,328 

(avg = 296) 

4.2 

(avg = 13.1) 


* The numbers of participants differs from the number recruited discussed in Chapter 3, as not everyone who initially agreed actually participated in 
the training. Source: MoS Employee Baseline Survey and Organizational Profile. 


In spite of the overall small proportional participant sample size, the demographic characteristics of 
participants receiving Measures of Success training as reported in the baseline employee survey were 
not dissimilar from those of the overall workforce in participating businesses as reported in the 
organizational profile. Nevertheless, Table 4.3 illustrates some interesting differences. First, there was 
a somewhat lower proportion of male participants receiving training relative to the proportion of men 
in the overall workforce in participating businesses (70 percent versus 80 percent in the overall 
workforce, respectively). Comparing businesses across provinces, several Nova Scotia-based 
businesses were in Social Services sectors with relatively few male participants [NS3, NS5, NS6, and 
NS7) while the opposite was the case in three Manitoba-based manufacturing sites [MB4, MBS, and 
MBS). 
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T able 4.3 Key Participant Demographic Differences compared with Business’s Workforces 


Characteristic - % distribution and incidence 

Participants 

Business’s 

Workforce 

Gender 

Femaie 

28.81 

19.8 

Maie 

70.41 

80.2 

Age (years) ^ 

Less than 30 

13.3(14.7) 

13.0 

30 to 40 

20.8 (23.0) 

16.0 

Above 40 

56.2 (62.3) 

71.0 


other 


Engiish as a second ianguage (ESL) 

13.3 (14.53) 

37.4 

Aboriginai 

3.5 (4.3^) 

4.2 

Totai 

226 

5,328 


Source: Measures of Success Baseline Participant Survey and Organizational Profile. 

^ 2 missing values; calculations in brackets exclude: ^ 22 missing values. ^ 9 missing values. “ 44 missing values. 


Second, participants were much younger, with a significantly higher proportion of participants 40 years 
of age and under compared to the business's total workforce. In just three businesses (NSl, MBS, MBS) 
was the share of participants 40 and under similar to the business workforce share; a significantly 
lower proportion of participants 40 and under was found in NS7 and NS8 (where over 90 percent of 
participants are above 40 years of age), while a significantly higher proportion of younger participants 
was found in MBl, MB13, and MB14. 

Third, there was a considerably lower proportion of ESL employees participating in the training 
compared to their share in the overall workforce of each business (a difference of about 24 percentage 
points). No Nova Scotia participants reported having ESL workers, whereas 29 percent of participants 
in Manitoba businesses did (not shown in the table). MBS, MB13, and MB14 each has a particularly high 
incidence at over 50 percent of participants reporting ESL, which may introduce additional challenges 
for ES training that are not present in Nova Scotia businesses. In regard to Aboriginal representation, 
participants were very similar to the overall workforce of the participating businesses. 

4.4 Lifecycle Traits: Marital Status, Dependents, Household Income 

Table 4.4 indicates that about 70 percent of participants were married or in a common law 
relationships which is somewhat higher than the proportion of working-aged Canadians overall 
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(61 percent^). Ignoring missing values, about half had children under 18 years old and still living at 
home, with this much more likely to be the case in Manitoba than in Nova Scotia. About 75 percent of 
participant households lived under the Canadian median household income of about $70,000,^ but 
considerably fewer in Manitoba than in Nova Scotia. 

Table 4.4 Participant Demographics: Martial Status, Dependents, and Household Income 


Characteristic - % distribution and incidence 

Total 

Nova Scotia 

Manitoba 

Married/common law^ 

70.8 

67.3 

74.0 

Have children under 18 years of age^ 

40.3 (51.2) 

43.0 (46.0) 

37.8 (60.0) 


Household income in the last 12 months, before taxes and other deductions^ 


Less than $30,000 

13.7 (20.8) 

18.7 (23.2) 

9.2(17.4) 

$30,000 to less than $40,000 

10.6(16.1) 

15.0(18.7) 

6.7(12.7) 

$40,000 to less than $50,000 

10.2(15.5) 

14.0(17.4) 

6.7(12.7) 

$50,000 to less than $70,000 

14.2 (21.5) 

19.7 (24.5) 

8.4(15.9) 

Greater than $70,000 

17.3 (26.3) 

13.1 (16.3) 

21.0 (39.7) 

Refused or Don’t know 

34.1 

19.6 

47.1 

Sample size 

226 

107 

119 


Source: Measures of Success Baseline Participant Survey. 

' 8 missing values. 2 51 missing values. 277 missing values. Numbers in brackets show percentages based on a sample size excluding those with 
missing values (non-responses). 


4.5 Employment Characteristics 

Data on employment characteristics were gathered to get an idea of participants' employment status 
(permanent/temporary, full/part-time), their wages, and their tenure with the business. These provide 
baseline measures and contextual variables relevant to future changes in labour market outcomes. For 
example, changes in employment status, hours of work, and wages are key outcomes of interest that 
may be directly influenced by training and therefore require pre- and post-program measures. At the 
same time, these characteristics also provide relevant contextual variables for which changes in other 
outcomes of interest may vary along. 

As Table 4.5 indicates, almost all participants were in permanent jobs (90 percent) and working full- 
time, i.e., 30 hours per week or more (96 percent), with small differences between provinces. 

Source: Statistics Canada, Canadian Census 2006. 

5 Actually it is $68,410. Source: Statistics Canada, CANSIM, table 111-0012. 
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Compared to provincial benchmarks, ^ permanent job incidence (figures in italics in table) was 
somewhat higher in the Nova Scotia sample, and about the same in Manitoba; the full-time job 
incidence rate for the sample is comparable to the rates in both provinces bearing in mind that the 
sample was predominantly in manufacturing where full-time incidence tends to be higher. 

The average job tenure of participants was close to 10 years, with half the participants working for 
their business for five years or more. Tenure was higher among employees in Nova Scotia businesses 
than in Manitoba, as indicated by the comparisons of the average [5 years difference) and the 
proportion with 20 or more years of tenure [about 24 versus 6 percent). Moreover, almost 50 percent 
of participants had been in their current position for three or more years, a proportion also higher in 
Nova Scotia than in Manitoba than. These results would suggest a fairly stable workforce, particularly 
in Nova Scotia, with a strong likelihood of remaining with the business for the duration of the training 
as well as a period after the training which facilitated post-program follow-up research. Comparing 
individual businesses, a higher proportion of participants had long tenure in several Nova Scotia 
businesses [NS5, NS7 and NS8), while a much lower proportion had long tenure in two Manitoba 
businesses [MBl and MBS). 


Table 4.5 Participants’ Employment Characteristics 


Characteristic - % distribution and incidence 

Total 

Nova Scotia 

Manitoba 

Permanent job 

90.3 

88.8 

(84.5&85.1)x 

91.6 

(96.9 & 87.6)x 

Full-time job 

96.0 

94.4 

(94.7 & 79.2}^ 

97.5 

(95.9 & 77.6) 


Tenure: number of years working with business** (17 missing) 


Average 

9.8 

12.2 

7.3 

Less than 1 year 

6.2 

6.5 

5.9 

1 to less than 3 years 

17.7 

22.4 

13.5 

3 to less than 5 years 

18.6 

12.2 

24.4 

5 to less than 10 years 

17.3 

15.9 

18.5 

10 to less than 20 years 

18.1 

18.7 

17.7 

20 years or more 

14.6 

24.3 

5.9 


Labour Force Survey results were obtained and computed from Labour Force Flistorical Review, 
Statistics Canada, 2009. 
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Number of years working in current position** (56 missing) 


Average 

7.5 

8.6 

5.4 

Less than 1 year 

8.9 

6.5 

10.9 

1 to less than 3 years 

20.4 

25.2 

16.0 

3 to less than 5 years 

10.6 

14.0 

7.6 

5 to less than 10 years 

13.7 

21.5 

6.7 

10 years or more 

21.7 

32.7 

11.8 

Held other positions with this employer** 

(24 missing) 

28.3 

33.6 

50.5 


Houriy wages** (57 missing) 


Average ($) 

18.43 

17.14 

(19.00)x 

20.70 

(19.88)x 

less than $15 

22.6 

32.7 

13.5 

$15 to less than $20 

26.6 

38.3 

16.0 

$20 to less than $25 

13.3 

21.5 

5.9 

$25 and greater 

10.6 

5.6 

15.1 

Sample size* 

226 

107 

119 


’^Provincial benchmarks (Labour Force Survey) in italics in brackets. Where there are two numbers in brackets, the first is the provincial average for 
manufacturing and the second for services. 

* Sample sizes vary due to missing values. 

“ Calculations exclude missing values. 

Source: Measures of Success Baseline Participant Survey. 


The average hourly wage of participants was $18.43, which somewhat lower than the Canadian average 
of $22.72.^ About 40 percent of participants were paid an hourly wage of between $15 and $25 per 
hour. In spite of longer tenure of participants in Nova Scotia, the average hourly wage of project 
participants tended to be higher in Manitoba. Four businesses stand out as having participants with 
high proportions with hourly wages less than $15 an hour (MB5, NS02, NS6 and NS7), while at least 
four others stand out with no participants earning less than $15 per hour.® At the same time, fewer 
participants in Nova Scotia businesses appeared to change positions over time, with only about a third 
having held previous positions with their employer compared to over half in Manitoba businesses. 
Differences across businesses are significant in this respect, suggesting a greater degree of lateral 
and/or vertical movement within several businesses, particularly in Manitoba (MB5, MB15 and NSl). 

^ Labcur Feme Survey scurces: Statistics Canada, CANSIM tables 282-0069 and 282-0073. 

® Other businesses aisc have nc participants earning less than $15 and heur but did net “qualify” as 
significantly Icwer because of high non-response rate for these businesses. 
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Looking at participant occupations (Table 4.6) reveals that recruitment for the project resulted in a 
wide range of positions that are clearly aligned with the sector of their employer. There are care 
workers from the two social services businesses (NS3 and NS5) and labourers, various production 
workers, lead hands and engineers from the other businesses, which are all in manufacturing. The wide 
range suggests potential wider applicability of the results beyond the 18 business participating in this 
project. 

T able 4.6 Occupation of Participants 


Code 

Occupations 

NS1 

Muitipie iabourer positions 

NS2 

Muitipie iabourer positions 

NS3 

Care workers 

NS4 

Labourers 

NS5 

Care workers 

NS6 

Supervisors and Front-iine staff hotei workers* 

NS7 

Front-iine food processing 

NS8 

Muitipie iabourers 

MB1 

Engineers 

MB2 

Dispatchers 

MBS 

Shop floor and iead hands 

MB4 

Shop floor and iead hands 

MB5 

Production workers 

MBS 

Production/iine staff 

MB11 

Supervisors 

MB13 

Production staff 

MB14 

Draftspersons, iead hands, shop floor 

MB15 

Team ieads 


* Training being delivered to each group separately. 

Source: Measures of Success Baseline Participant Survey and administrative data. 
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4.6 Educational Credentials of Participants 

A key component of human capital is the level of prior education and credential attainment. To 
determine the education level, participants were asked about all their educational diplomas, certificates 
or degrees attained in the past, prior to the project. 

The results presented in Table 4.7 indicate that approximately 1 in 4 participants had only a high 
school diploma, not dissimilar from the Canadian average. However, about 1 in 6 participants have no 
credential at all [less than high school) compared to about 1 in 10 among the Canadian population.^ At 
the same time, about 56 percent of participants had some form of a post-secondary credential, which is 
lower than the Canadian labour force [about 62 percent). On the other hand, the proportion of 
participants with a non-university PSE credential [about 46 percent) was somewhat higher than the 
Canadian labour force [36 percent). 

Overall, Nova Scotia participants had lower levels of educational attainment than Manitoba 
participants. Nova Scotia sites had higher percentages of participants with no educational credential or 
with only a high school diploma, and lower proportions with a post-secondary credential. While there 
was a higher percentage of participants with trade or vocational certificates in Nova Scotia than 
Manitoba [28 versus 14 percent respectively), there were higher percentages of participants with a 
college diploma and with a university degree in Manitoba [47 versus 20 percent respectively). 

Two businesses accounted for the high proportion of university graduates in Manitoba, MBl 

[80 percent) and MB13 [56 percent). Two businesses also had participants with significantly greater 

proportions without credentials than the other businesses [NS7 and MB14). 

Table 4.7 Participants’ Highest Educational Credentials (%) 


Credential 

Total 

Nova Scotia 

Manitoba 

No credentiai 

15.9 

20.6 

11.8 

A high schooi dipioma oniy 

25.7 

29.0 

22.7 

A post-secondary credentiai (highest*): 

56.2 

49.7 

62.2 

Apprenticeship dipioma 

1.3 

1.9 

0.8 

Trade/vocationai dipioma or certificate 

20.8 

28.0 

14.3 

Coiiege dipioma 

23.5 

17.8 

28.6 

University degree 

10.6 

1.9 

18.5 

Non-response 

2.2 

0.9 

3.4 

Sample size 

226 

107 

119 


* An individual can have more than one post-secondary education credential, but the figures were re-computed to show only the highest. The order 
of precedence assumed is as in indicated in the table, from apprenticeship up to university, as per Statistics Canada practice. 

Source: Measures of Success Baseline Participant Survey (n=225). 


Labour Force Survey, 201 1 , CANSIM, Statistics Canada: httD://www5.statcan.qc.ca/cansim/a26 . 
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PART II: Changes in Participant and Business Outcomes 
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Chapter 5; Intermediate Outcomes - Human, Psychological, 
and Social Capital 

This chapter presents results for key intermediate outcomes of interest that may be affected fairly 
quickly following training. These include human capital and skills [e.g., Essential Skills), associated 
literacy practices (e.g., everyday use of literacy skills), psychosocial outcomes [e.g., motivation, self- 
confidence, work-related stress), and some indicators of social capital [e.g., improved social networks). 
These outcomes can both mutually reinforce and also support the longer-term outcomes of interest 
discussed in the next chapter. 

Highlights of key results are presented here followed by a review of the methodology used to assess 
changes in these outcomes. Each section illustrates how the sample changed from baseline to the post- 
training follow-up, with a focus on results at 6-months. 3-month results are discussed when they differ 
substantively. While changes in job performance are often considered intermediate outcomes of 
training, these are addressed in the following chapter along with changes in business results and other 
long-term outcomes of interest. 

5.0 Highlights 

■ While the majority of participants had a high level of confidence in their literacy skills at 
baseline, a significant minority lacked confidence in a work setting, leaving significant room 
to improve. Confidence was highest in reading skills with about 80 percent feeling their skills were 
adequate to do their job well. This was followed by numeracy and writing skills with about three 
quarters expressing confidence. These results are somewhat lower than national levels in the lALSS 
sample. Baseline levels also suggest a significant minority of participants had little confidence in a 
work setting and had substantial room for improvement. 

■ Changes observed through 6-months indicate that confidence in work-relevant literacy skills 
such as numeracy and technical skills have improved significantly. Increased confidence in 
everyday literacy skills such as reading, writing, math, and digital technology were also observed. 
There were significant post-training changes in three different math indicators - including 
increased confidence in doing math, being good with figures and calculations, and reduced anxiety 
associated with figuring out amounts. 

■ While most participants have social networks that provide access to multiple sources of 
support, a significant percentage had quite small, dense and homogeneous networks at 
baseline. Over 40 percent of participants had 3 or fewer contacts for various kinds of bonding and 
bridging supports and nearly two-thirds have fairly dense networks. The combination of small and 
dense social networks, with limited bridging social capital, was a significant limiting factor for some 
participants in terms of options for engagement within their communities, limiting their 
opportunities for further learning and use of literacy skills. 
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■ Significant improvements were observed in social capital following training, particularly 
among those with low levels at baseline, including increases in network size and improved 
composition. In addition to gains in network size, participant networks also expanded in terms of 
breadth and types of support. For example, a substantial number of participants went from closed, 
completely interconnected networks, to having less dense networks, which are better for 
leveraging other kinds of resources and supports, including further learning opportunities. 

■ There were few significant gains in indicators of psychological capital including motivation 
and engagement at work and attitudes towards continuous learning. This may in part relate to 
the fact that very high levels of self-efficacy were observed at baseline among this sample and many 
indicators were subject to ceiling effects.io Some exceptions include improvements in the incidence 
of job-related stress. 

5.1 Methodology 

Quantitative analyses were conducted to investigate whether participants reported improvement after 
six months. For each outcome of interest, formal tests of statistical significance were carried out in 
order to assess both: 

a] The direction of change, i.e. whether a significantly greater proportion of participants reported 
gains than losses. 

b) The magnitude of change, i.e. whether the average net gain for all participants was significantly 
greater than zero 

Of 77 outcomes examined, 18 met one or both of the above criteria and were statistically significant. 
These are highlighted below. These outcomes are organized by key domains — those relevant to 
human capital [e.g., skills and practices relevant to both work and everyday life), psychological capital 
(e.g., motivation and engagement and work-related stress), social capital [e.g., network size, 
composition), and longer-term non-financial at the participant-level related to inclusion, cohesion and 
wellbeing [e.g., trust and satisfaction with work and life) and both tangible and intangible business 
outcomes. 

5.2 Literacy Skills and Practices 

Skills Relevant to Work 

Baseline results suggest that the majority of participants had a high level of confidence in their literacy 
and technical skills at work . Between 74 and 80 percent agreed or agreed strongly they had each of the 
literacy skills they need for the job, and 76 percent felt that way about their technical skills. Less than 
6 percent of participants disagreed or disagreed strongly with each of the skills statements. 

A ceiling effect occurs when as part of a baseline survey a high proportion of respondents provide 
answers at the highest possible category (for instance, “I am very confident in my writing skills”), leaving 
little or no room for gains for a large portion of the sample in subsequent surveys. We provide a more 
thorough explanation 
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Nonetheless, nearly 1 in 5 participants responded neutral to these questions, particularly about their 
math and technical skills at work, an indication that there may be modest room for improvement in 
these areas. 


Post-training, learners reported significant gains in confidence in these precise areas, specific to math 
and technical skills for their main job. Figure 5.1 shows the percentage of workers who agreed or 
strongly agreed that they had the math skills required to do their job well rose from 79% at baseline to 
86% at 6-months. Similarly, Figure 5.2 illustrates that the percentage of workers who agreed or 
strongly agreed that they had the technical skills required to do their job well increased from 79% at 
baseline to 87% at the 6-month follow-up. 


Figure 5.1 Percentage of Participants with Confidence in Math Skiiis, at Work 



strongly Disagree/ Meutral Agree Strongly Agree 

Disagree 


■ Base line % 

■ 6 mo % 


Responses to "I have the math skills I need to do my main job well" 


Figure 5.2 Percentage of Participants with Confidence in Technicai Skiiis, at Work 
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48% 



■ Baseline % 

■ 0 mo% 


Responses to "I have the technical skills I need to do my main job well" 
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However, there were no gains in reading or writing skills needed to do the main job well, or in an array 
of workplace practices involving use of these skills. This pattern of results for skill gains at work at 6- 
months mirrors the early results that were observed at the 3-month follow-up. 


Literacy Skills and Practices in everyday life 


Baseline results suggest that confidence in using literacy skills at home in everyday life [not workplace 
specific) was lower than at work, with some measures indicating that only a minority of participants 
were confident in their skills. Overall, participants appeared more anxious making numerical 
computations than the labour force at large, based on a comparison of these results to the lALSS. They 
were also more likely to agree that they read only when they had to and were less likely to agree that 
they enjoyed talking about what they read. 


Post-training, participants reported a wide range of gains in the context of everyday skills - specifically, 
improvements in confidence to use literacy skills, gains in frequency of practice of such skills, and more 
positive attitudes to activities associated with the practice of literacy skills. A larger proportion of 
learners at 6-months than at baseline reported that they were very confident in their abilities to do 
math [60% vs. 64%, Figure 5.3), read non-fiction [49% vs. 58%, Figure 5.4), write [40% vs. 51%, 

Figure 5.5), and use a computer [43% vs. 56%, Figures 5.6). 


Figure 5.3 Percentage of Participants with Confidence in Math Abiiity 
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Figure 5.4 Percentage of Participants with Confidence in Reading (Non-Fiction) 



■ Baseline % 

■ 6 mo% 


Figure 5.5 Percentage of Participants with Confidence in Writing 
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Figure 5.6 Percentage of Participants with Confidence in Computer Use 



In terms of everyday practice of literacy skills, the proportion of learners who reported reading fiction 
at least a few times per week increased from 22% at baseline to 30% at 6-months (Figure 5.7). 
Similarly, the proportion of learners who reported reading non-fiction at least a few times per week 
increased from 50% at baseline to 56% at 6-months (Figure 5.8). Those who reported visiting a library 
or bookstore at least once per week rose from 7% to 10%, while those who reported never visiting 
decreased from 26% to 22% (Figure 5.9). 

Figure 5.7 Frequency of Reading, Everyday Practice 
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Never Rarely <l/wk 1/wk Afew Every day 

times/wk 

Responses to "How often doyou read non-fiction?" 


■ Baseline % 

■ 6 mo% 


Figure 5.9 Frequency of Visiting a Library 



Responses to "How often do you use a library or visit a bookstore?" 


■ Baseline % 

■ 6 mo % 


Finally, a larger proportion of learners at 6-months than at baseline reported the positive attitude that 
they were good with numbers and calculations, and a smaller proportion of learners reported the 
negative attitude that they were anxious when figuring out amounts (e.g. discounts, sales tax or tips). 
Those who agreed or strongly agreed that they were good with numbers rose from 63% at baseline to 
74% at 6-months (Figure 5.10); those who disagreed or strongly disagreed fell from 9% to 5%. Those 
who disagreed or strongly disagreed that they were anxious when figuring out amounts rose from 47% 
at baseline to 57% at 6-months [Figure 5.11). 
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Figure 5.10 Confidence in Numbers and Caicuiations 



strongly Disagree Neutral Agree Strongly 
Disagree Agree 


Responses to "I am good with numbers and calculations" 
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Figure 5.11 Anxiety in Using Numeracy Skiiis 



strongly Disagree Neutral Agree Strongly 
Disagree Agree 


Responses to I feel anxious when figuring out amounts 


This pattern of results for skill gains in everyday life at 6-months is similar to the preliminary pattern of 
results that was reported at 3-months. At both 3- and 6months, learners showed gains in confidence in 
their abilities to read non-fiction, write, and use a computer; in how often they read fiction, and visited 
a library or bookstore; and in the positive attitude of feeling that they were good with numbers and 
calculations. In addition, at 6-months a few results emerged that had not been observed at 3-months: 
Learners at 6-months showed gains in confidence in their ability to do math and how often they read 
non-fiction, as well as reductions in the negative attitude of feeling anxious when figuring out amounts. 
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However, a few results that had emerged at 3-months were not seen at 6-months: Learners at 6-months 
no longer showed gains in confidence in their ability to read fiction, how often they did math [such as 
for bills, bank accounts or credit cards) or in the positive attitudes of feeling that reading is one of their 
favourite activities or enjoying talking about what they’ve read with other people. 

5.3 Psychological Capital - self-efficacy, motivation, and work-related stress 

Baseline results indicate that most participants had a fairly high level of engagement and motivation in 
regard to their work. Across most measures, at least 4 in 5 participants agreed or strongly agreed with 
the statement that suggests they were highly motivated and engaged in their jobs. For example, nearly 
85 percent of participants agreed or strongly agreed that they did a good job, that what they did at 
work was important, that they tried to plan what they do at work, and that they persisted in their job 
when it was challenging. The latter indicator can be seen as another positive measure of resilience. 
Similarly, participants appeared to have a high-level of self-efficacy in their resourcefulness and their 
ability to solve problems. For all measures but one, at least four in five participants believed the 
statement indicating self-efficacy is moderately or exactly true. 

Post-training, there were few gains in psychosocial outcomes from baseline to 6-months, likely related 
to the high level of positive responses at baseline [see more on ceiling effects in Chapter 8 on data 
quality). Nevertheless, the modest reductions in work-related stress that were apparent immediately 
after training were maintained at 6-months. For example, the significant decrease in job anxiety levels 
shown at 3-months was maintained at 6-months. The proportion of respondents who agreed or 
strongly agreed at baseline that they felt anxious fell from 31% to 28% at 6-months, while the 
proportion who strongly disagreed almost doubled from 6% to 11% [Figure 5.12). 

Figure 5.12 Feelings of anxiety in one’s job 
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5.4 Social Capital - Network Size and Composition 


Baseline results illustrate that nearly 40 percent, a large percentage, of participants had very small 
social networks, with 3 or fewer contacts to call on for various kinds of support. Compared to a national 
sample of respondents to the Canadian General Social Survey (GSS), the percentage of participants with 
small networks was quite high, suggesting that many were fairly isolated. In fact, a small but significant 
proportion of participants [11 percent) had no sources of assistance to call upon in different aspects of 
their life prior to training. Small networks reduce not only social interaction, and the benefits they 
bring, but also limit opportunities to participate in further learning and other valued dimensions of life 
— community, economic, or political. 

Post-training, the total size of participants' networks, as well as the number of contacts they had for 
specific types of resources, such as household supports, increased significantly both at 3-months and in 
the longer-term at 6-months. Those who reported a total network size of zero fell from 11% at baseline 
to 3% at 6-months, while those with a total network size of 7 or more contacts rose from 33% to 43% 
[Figure 5.13). In terms of specific kinds of help, those who reported three or fewer contacts who they 
could call upon for help with household activities - such as childcare, household maintenance, and 
personal care - fell from 60% at baseline to 49% at 6-months [Figure 5.14). There was also a gain in the 
number of contacts who could help with job or career support observed at 3-months but was no longer 
present at 6-months. 

Figure 5.13 Social Networks: Total Number of Contacts 
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Figure 5.14 Social Networks: Number of Contacts for Household Supports 



Network Density and Diversity 

In addition to having small social networks, a significant percentage of participants had quite dense and 
homogeneous networks prior to training. Over two-thirds of participants reported that all or most of 
their contacts knew each other, and that over half report all or most lived within the same community. 
The combination of small and dense social networks with limited bridging social capital is a significant 
limiting factor in terms of options for engagement within their communities, and limiting their 
opportunities for further learning and use of literacy skills. 

Post-training, there were significant changes in network composition from baseline to 6-months. In 
terms of network density there was movement from each of the extremes towards the middle of the 
distribution. Those who had highly interconnected networks at baseline, in which everybody knew 
everybody else, started meeting people outside their regular social circles (thus building bridging 
social capital), so that at 6-months, most, rather than all, of their contacts knew each other. At the 
other extreme, those with highly disconnected networks at baseline in which hardly anybody knew 
anybody else began to build some interconnectedness [bonding social capital) within their networks, 
so that at 6-months, some, rather than few or none, of their contacts knew each other (Figure 5.15). 
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Figure 5.15 Network Composition: Density of Contacts 
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There was also movement away from the extremes towards the middle of the distribution in terms of 
the diversity of networks. Some participants expanded their social network beyond work, moving from 
knowing all or most of their contacts from work to knowing only some of their contacts from work. 
Others developed a few contacts at work where they had previously had none (Figure 5.16). In both 
cases, participants expanded the diversity of their networks. 


Figure 5.16 Network Composition: Diversity of Contacts, by Workpiace 
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A similar trend was shown in number of contacts with a different occupation. Some participants 
expanded the diversity of their networks by meeting people within their own occupation, moving from 
having all of their contacts with a different occupation to most or some of their contacts with a different 
occupation. Others met people from different occupations, moving from no contacts with a different 
occupation to few or some [Figure 5. 17). 


Figure 5.17 Network Composition: Diversity of Contacts, by Occupation 
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Chapter 6: Longer-term Participant and Business Outcomes 

This chapter reviews changes in longer-term outcomes of participants and businesses following the 
Measures of Success training intervention, as measured at the final 6-month follow-up. For participants, 
this includes financial outcomes such as wages and earnings. It also includes non-financial measures of 
important social indicators such as well-being (e.g., satisfaction with work and life), work-family 
balance, social cohesion [e.g., sense of trust and belonging), and social inclusion [e.g., participation in 
community). 

The chapter also reviews changes in worker performance and the related effects that these can have on 
business outcomes such as productivity, costs, health and safety, and sales. A range of intangible 
business outcomes is also considered, such as employee-management relations, a sense of unity among 
staff, workplace participation, and the extent of a learning culture. The range of measures used allows 
for a rich illustration of both the tangible and intangible performance and business outcomes that may 
be influenced by Essential Skills training. These are also the ones that were of most interest to the 
businesses that participated in this project. 

6.0 Highlights 

■ Several indicators of improvements in participant well-being were observed including 
increases in satisfaction with life, work-family balance, and various indicators of job 
satisfaction. This included reduction in job anxiety, improvement in the ability to balance the 
demands of work and family, and increased satisfaction with the overall quality of working life. No 
accompanying gains in financial outcomes for participants were observed, as 6-months may be too 
short a time frame to capture changes in wages or earnings. 

■ Positive changes were also observed in at least one important indicator of social cohesion, 
with a significant increase in trust among participants in their more distant contacts. 

Participants' levels of trust in close connections such as neighbours, co-workers and supervisors 
remained high throughout the follow-up. However, post-training, they also broadened their spheres 
of trust to include more distant connections, which may facilitate longer-term development and use 
of social networks leading to more opportunities for literacy practice and skill development. 

■ A wide range of performance gains was reported in areas related to task efficiency and 
accuracy as well as interpersonal relations including better communication with co-workers 
and customers. Between half and three quarters of participants agreed or strongly agreed that 
they had improved in each of the key performance areas over the 6-months since training onset. Of 
those reporting improvement, about three-quarters attributed at least some of the change directly 
to training. 

■ Employers reported training-related performance gains in a range of areas consistent with 
their business needs and prior expectations on both tangible and intangible outcomes. At 

least half of participating employers reported training-related improvements in productivity, 
reductions in errors and costs, and better employee-management relations. Gains were reported by 
an increasing number of employers and a broader range of outcomes over time. 
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6.1 Well-being - Satisfaction with Work and Life 


At baseline, when asked about their satisfaction with various aspects of their work, a significant 
majority of participants reported being content with the quality of their working conditions. For 
example, 80 percent reported having the opportunity to use their abilities to do their job, three- 
quarters said they worked in a safe environment, and two-thirds were satisfied with their working 
conditions. Two thirds of participants also reported being satisfied with their life in general. 

In spite of fairly high baseline levels of satisfaction with work and life, post-training results revealed 
modest improvements in a number of key job satisfaction indicators. For example, the proportion that 
agreed or strongly agreed that they were satisfied with their overall quality of working life rose from 
67% at baseline to 71% at 6-months [Figure 6.1). In addition, the proportion that strongly agreed that 
their employer provided adequate facilities and flexibility for work-family balance rose from 24% to 
31% at 6-months [Figure 6.2). Significant gains were also observed in overall life satisfaction at the 3- 
month follow-up although these were no longer statistically significant at the final follow-up at 6- 
months. 

Figure 6.1 Satisfaction with overaii quaiity of working iife 
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Figure 6.2 Satisfaction with the flexibiiity one’s empioyer provides around famiiy life 



strongly Disagree Neutral Agree Strongly 
Disagree Agree 


■ Baseline % 

■ 6 mo % 


Responses to "My employer provides adequate flexibility for me to fit work in 

around my famiiy life" 


6.2 Social Cohesion - Measures of Trust 

Social cohesion constitutes the perceptions of individuals about their place in a community or 
organization, e.g., perceptions of a shared set of values, a sense of belonging and reciprocity among 
members, and, particularly, the trust they have for each other. Workers who feel they are part of a 
community, with a sense that co-workers will help one another, and who have the trust of colleagues 
and management, are not only more likely to have job satisfaction, but also to be stable, productive, 
employees. 

Trust is an important indicator of social cohesion among groups, which can facilitate networking, social 
interaction, and particularly, teamwork, in a business setting. Measures of trust often involve assessing 
perceptions about the "trustworthiness" or likelihood of fellow members of an organization taking a 
selfless action. One of the more common measures employed in research on cohesion involves 
perceptions of the likelihood that a lost wallet containing money would be returned if found by 
different members of a community. These measures can be applied to different settings or groups 
including work colleagues [coworkers, supervisors), neighbours [primary, secondary), civil service 
[police, fireman) or total strangers. 

At baseline, participants had a fairly high level of trust among their colleagues, which in fact was 
highest at work, compared even to the trust they had among their home neighbours. Notably, 
participants trusted their supervisors more than their co-workers, with 79 percent perceiving 
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supervisors as trustworthy compared to 58 percent for co-workers. Less than half of participants 
identified strangers as trustworthy, indicating that there could be significant room to broaden the 
sphere of trust to more distant connections, which would facilitate further networking and 
opportunities for learning and skill development. 

Post-training, modest yet significant gains in two particular vectors of trust - towards more distant 
network connections - had developed at 6-months, which were not present immediately after training 
at 3-months. Figure 6.3 illustrates that the proportion of participants who identified a local grocery 
clerk as trustworthy increased from 30% at baseline to 39% at 6-months. Similarly, Figure 6.4 indicates 
the proportion of participants identifying strangers as trustworthy increased from 48% at baseline to 
52% at 6-months. 

Figure 6.3 Trust in More Distant Connections - Locai Cierk 
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Figure 6.4 Trust in More Distant Connections - Strangers 



6.3 Job Task Performance - changes in the way participants work 

In the 6-month survey, participants were asked a series of retrospective questions on changes in the 
way they did specific parts of their jobs since the baseline interview. In an effort to further support the 
attribution of performance gains to the training, participants were also asked about the likelihood that 
the change would have happened without training [i.e., to provide an "implicit counterfactual"). The 
results are presented in Figure 6.5 and summarized below. 

For most measures of job performance, about 40-50% of participants reported both improvement over 
the 6-month period, and attributed at least some of that improvement to the training they had 
undertaken. Measures for which participants were most likely to report training-related improvement 
included communicating and working effectively with co-workers, understanding and using workplace 
documents, and organization and time management - for all of which more than 50% of participants 
both reported improvement and attributed at least some of that improvement to training. 

Measures for which participants were least likely to report training-related improvement include 
ability to use workplace equipment, encouraging customers to buy company products and services, and 
ability to use computers and work-related software - for all of which 40% or fewer of participants both 
reported improvement and attributed at least some of that improvement to training. 
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Figure 6.4 Changes in Job Task Performance: % of participants reporting gains at 6-months 
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6.4 Changes in Business Outcomes, at 3- and 6-months Post-Training 

At baseline, while senior managers and supervisors tended to rate their businesses fairly high in most 
areas, a significant majority gave one or more key business areas lower ratings, which indicated room 
for possible improvement. In addition, most employers were optimistic about the chances of the 
Essential Skills training improving the key business results of interest to them. A significant majority of 
senior managers interviewed [77 percent) reported that they expected the training to possibly or 
definitely improve all of the key business results of interest. 

For the kinds of changes expected by businesses, more than 90 percent of managers expected the 
training to increase productivity, reduce costs, improve customer service, and increase sales. All but 
three tangible outcomes — health and safety, turnover, and absenteeism — were expected by most 
managers to improve. These were also the areas about which supervisors expressed the least amount 
of optimism, which suggests they are the areas on which we should least expect training to have effects. 

With respect to intangible outcomes, nearly all project sponsors reported positive expectations that 
training would improve management relations, employee involvement, and the degree of learning 
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culture. Supervisors were also optimistic, although less so than project sponsors, that the training 
would positively affect these intangible outcomes. 

At the 3- and 6-month post-training follow-ups, senior managers and supervisors in each of the 
eighteen participating businesses were each asked about possible improvements in eleven different 
business outcomes covering both the tangible and intangible areas identified at baseline [for a total of 
198 outcome responses). Employers were also asked for their degree of certainty that these changes 
were related to the training intervention, either possibly, or definitely. 

At the 3-month follow-up, training-related improvement was reported as possibly related to training in 
nearly a third of businesses [59, or 29.8% of the cases). Figure 6.5 illustrates the responses and the 
degree of certainty that employers felt these were attributable to training. Improvements were 
generally more prevalent among intangible outcomes [41.7%) than among tangible outcomes [23.0%). 
For example, 8 out of 18 sponsors [44%) reported improvements in employee management relations 
and sense of unity /belonging, that they either definitively or possibly attributed to training - and 7 out 
of 18 [39%) reported training-related improvements in employee involvement/participation and 
learning culture. 


In terms of tangible outcomes, the two outcomes for which sponsors were most likely to report 
training-related improvements were productivity [10 out of 18, or 56%) and costs and errors [7 out of 
18, or 39%). Five out of 18 sponsors [28%) reported training-related improvements in product or 
service quality, while 4 out of 18 [22%) reported improvements in turnover and absenteeism. Only 
two sponsors reported improvements in customer service, one in health and safety and none in sales. 


Figure 6.5 Changes in Business Outcomes: % of Businesses reporting gains at 3-months 
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Figure 6.6 illustrates results from the 6-month post-training follow-up. Results suggest that sponsors 
reported improvements attributable to training in a greater number of business outcomes than they 
had reported at 3-months. Sponsors were each asked about possible improvements in the same eleven 
different business outcomes for a total of 198 outcomes. Training-related improvement was reported in 
72 [36.3%} of the cases [compared to 29.8% at 3-months). The increase in sponsor-reported 
improvements attributable to training from 3-months to 6-months was especially pronounced for 
intangible outcomes [51.4% at 6 months compared to 41.7% at 3-months) where an increasing number 
of employers report improvements in employee-management relations and the presence of a learning 
culture. 

Training-related improvement in tangible outcomes also increased slightly at 6-months compared to 3- 
months [27.7% compared to 23.0%) with an increasing number of employers reporting improvements, 
in productivity and costs and errors. Interestingly, improvements in several business outcomes 
emerged after 6-months, which were not present at 3-months, and/or with much more certainty 
among employers in their link to training, including health and safety and sales revenue. 

Figure 6.6 Changes in Business Outcomes: % of Businesses reporting gains at 6-months 
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Chapter?; Understanding Training Success Factors; 

Context, Characteristics, and Process 

Multivariate regression analysis is a statistical technique that allows us to explain the importance of an 
individual factor in a process by holding all the other factors constant. For Measures of Success, the 
technique allowed us to explore how individual and firm characteristics and the training context and 
implementation process were linked to changes in participant and business outcomes. We were able to 
address questions such as the following: If two people share most characteristics, differing only in level 
of education, which one is more likely to realize larger gains from training - the one with the higher or 
lower level of education? If two people work in firms that share the same context, differing only in 
employer expectations with regard to the training, which individual is more likely to realize higher 
gains from training - the one with higher or lower employer expectations? 

This analysis was a crucial part of the overall approach to confirming the link between the training and 
the gains described in earlier chapters. The analysis helps validate expectations regarding the pattern 
of results and, more generally, gives us a better understanding of the conditions and factors that can 
influence training success. We offer the explanatory analysis of key participant outcomes in the first 
section of this chapter followed by the analysis of performance and business outcomes. Each sub- 
section highlights the factor that was analyzed, followed by a discussion of the outcomes that they help 
explain. 

7.0 Demographic Characteristics and Participant Outcomes 

While post-training improvements are realized by a wide variety of sub-populations, some demographic 
groups gain more than others. 

Educational Attainment 

Improved outcomes extended not only to those with higher levels of educational attainment but also 
those with high school or below. Those with postsecondary education realized higher gains in five 
outcomes related to confidence and frequency of using literacy skills, trust, and well-being. Those with 
high school or less realized higher gains in social networks and in reducing anxiety associated with 
work. For the majority of outcomes, there was no significant difference between gains realized by those 
with higher and those with lower levels of educational attainment. 

English as a Second Language 

In terms of language use, those who used English as the main or only language at home realized higher 
gains in five outcomes, primarily in confidence in math and technical skills, and in trust in more distant 
contacts; those who regularly used languages other than English realized higher gains in literacy 
practices, notably infrequency of reading and using a library. 

Gender 

Men realized higher gains than women in four outcomes, related to confidence in math and technical 
skills, while women had larger gains in three outcomes related to confidence and frequency in reading 
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and reductions in work-related anxiety. However for the majority of outcomes, there was no link 
between gains and participant gender. 

Age 

In terms of age, older workers (45+) experienced wider gains associated with confidence in technical 
skills, frequency of literacy practices, social networks, and work-family balance, while younger workers 
[under 35) realized wider gains associated with literacy skills, job-related anxiety, and trust. 

7.1 Mediating Factors that influence Changes in Participant Outcomes 

1) Social Capital 

While gains were experienced both by those with high and low levels of social capital, those with larger 
networks prior to training experienced greater gains in a subset of outcomes. 

Those with larger networks realized higher gains in five outcomes related to confidence in literacy 
skills including writing, numeracy, and digital technology as well as improvements in work-family 
balance. However, for the majority of outcomes, there was no link between gains and network size. 

2) Psychological Capital 

Improvements from baseline to 6-months were more likely to be reported by those with low levels of 
psychological resources at baseline, though the difference was not as large as at 3-months. 

Higher gains were usually realized by those with greater levels of maladaptive cognitive issues at work 
[e.g. "I get quite anxious in my job”, "I find I sometimes reduce my chances of doing well in my job”), 
lower self-efficacy, and/or lower self-esteem. Participants with at least one of these characteristics 
realized higher gains in 8 outcomes, while participants with lower levels of maladaptive cognitions at 
work, higher self-efficacy, and/or higher self-esteem realized higher gains in only 3 outcomes [see 
below for details). 

This suggests that training may be at least as effective, or possibly more effective, for people who are 
less able to cope with the psychological challenges and stresses of work. However, while a wide range 
of gains was reported by this vulnerable population, there were few corresponding gains in 
psychological variables from baseline to 6-months [with the exception of a reduction in job anxiety). 

3) Attitudes towards Training 

Improvements from baseline to 6-months were more likely to be reported by those with positive attitudes 
towards the training, and those with a higher number of training-related goals. 

A broader range of improvements was seen among those who have positive attitudes about the training 
[i.e. those who agree or strongly agree with statements such as "I am looking forward to taking this 
training” and "I am motivated to do the best I can in this course”), and for those who could identify clear 
and specific goals in relation to the training [i.e. those who say that they were taking the training in 
order to advance in their job or career and/or to improve specific skill sets). 
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These individuals experienced larger gains in literacy practices including frequency of reading [non- 
fiction), frequency of visiting a library, confidence in math skills, and reductions in job anxiety. Those 
with positive attitudes also experienced larger improvements in social networks. Those with more 
specific training-related goals also realized higher gains in confidence in using a computer. 

Higher gains were not limited exclusively, however, to those with more positive attitudes or clarity in 
their training goals. Those with negative attitudes about learning in general [i.e. those who disagree 
with statements such as "You are more likely to get a better job if you do some learning” and agree that 
"Getting qualifications requires too much effort") also realized higher gains but in a more limited set of 
outcomes; these included confidence in writing, frequency of reading [fiction), and reductions in job 
anxiety. 

7.2 Implementation Factors that influence Changes in Participant Outcomes 

1) Training Hours, Location 

Participant gains from baseline to 6-months were more likely to be reported by workers in firms that 
delivered more hours of training 

In the multivariate analysis, a higher "dosage" of training was linked to higher participant gains in 
6 outcomes, while a lower dosage of training - fewer than 10 hours - was linked to higher gains in only 
1 outcome.il Higher gains were observed throughout a range of outcomes linked with human, social, 
and psychological capital. 

Specifically, firms that delivered a higher dosage of training realized higher worker gains in: 

■ Math skills needed to do main job well 

■ Confidence in ability to write 

■ Frequency of reading non-fiction 

■ Frequency of visiting a library or bookstore 

■ Trust in strangers 

■ Number of contacts to provide help with household activities 

Firms that delivered a lower dosage of training realized larger reductions in job anxiety. 


This result is somewhat confounded by the fact that all the firms delivering 30 or more hours of training 
were based in Nova Scotia, so it is not entirely clear whether the result represents a training “dosage 
effect” or a “province effect”. Nonetheless, given the similarity in the institutional delivery models of these 
long-standing workplace training programs, the higher dosage likely contributes substantively to these 
differences in outcomes. 
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Participant gains from baseline to 6-months were especially likely to be reported by workers from firms that 
delivered the training onsite 

The multivariate analysis allowed us to compare gains among sets of identical participants from firms 
that were identical in every respect but one - i.e. location of training delivery [onsite vs. offsite). 

Onsite delivery was linked to higher gains in 10 participant outcomes, while offsite delivery was linked 
to higher gains in only 1 outcome, measures of bonding social capital. In addition to higher gains in 
human, social, and psychological capital, onsite delivery was associated with improvements in overall 
work satisfaction and better work-family balance. 

Specifically, firms that delivered training onsite realized higher worker gains in: 

■ Confidence in ability to do math 

■ Confidence in ability to read non-fiction 

■ Confidence in ability to write 

■ Confidence in ability to use a computer 

■ Reporting that they are good with numbers 

■ Reducing anxiety associated with figuring out amounts 

■ Trust in strangers 

■ Total number of contacts to provide various kinds of help 

■ Overall work satisfaction 

■ Work family balance 

7.3 Firm Characteristics - Performance Incentives 

Improvements from baseline to 6-months were more likely to be reported by workers from firms that 
offered performance incentives 

Offers of performance incentives were linked to higher gains in 10 participant outcomes, while no 
performance incentives were linked to higher gains in only 1 outcome, measures of bonding social 
capital. 

Firms that offer performance incentives realized higher worker gains in: 

■ Math skills needed to do main job well 

■ Confidence in ability to do math 

■ Confidence in ability to read non-fiction 

■ Confidence in ability to write 

■ Frequency of reading non-fiction 
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■ Total number of contacts to provide various kinds of help 

■ Overall work satisfaction 

■ Work family balance 

Other firm characteristics that were expected to influence participant outcomes - including prior 
experience with Essential Skills training, employer expectations, and whether the training was 
voluntary or mandatory - were associated with more mixed results or results whose effect was limited 
in time, observed only at the 3-month follow-up but not the 6-month. 

7.4 Explaining Differences in Performance 

Job performance outcomes differ from the other participant outcomes discussed above in that they do 
not rely on baseline and follow-up questions to determine pre-post training differences. Instead, 
retrospective questions, asked only in the 6-month follow-up survey, were used to determine pre-post 
improvements in specific job areas, including productivity, organization, technical skills, emergency 
preparedness, and relations with co-workers and customers. 

For the other participant outcomes, multivariate regression analysis allowed us to explore how 
individual and firm characteristics as well as other implementation factors were linked with 
improvements in job performance, reported retrospectively at 6-months. 

The analysis revealed a number of important factors that can help explain improvements specifically in 
job performance. 

7.5 Demographic Differences and Job Performance 

In general, improvements in job performance were reported by a wide variety of sub-populations, especially 
those wbo regularly use a language other than English at home. 

Larger improvements were reported in different job performance areas by both those with higher 
levels of educational attainment and those with high school or below. Those with less education 
reported more improvement in two areas including working with workplace documents and workplace 
equipment. Those with more education reported more improvement in one area, being better able to 
communicate with customers. For the majority of outcomes, there was no significant difference 
between improvements reported by those with higher and lower levels of educational attainment. 

In terms of language use, those who regularly used languages other than English reported more 
improvement than those who use English as the main or only language in most job performance areas. 

Those who regularly used languages other than English at home reported more improvement in 
their ability to: 

■ be faster at getting things done 

■ complete tasks more accurately, with fewer errors 

■ be more organized and able to plan their time more effectively 
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■ understand and use workplace documents more easily, such as workplace manuals or reports 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ use workplace equipment more effectively, such as tools for my job or office machines 

■ communicate and work more effectively with co-workers, such as completing different job tasks in 
teams 

■ communicate with customers, such as describing features and benefits of the company's products 

They were also more likely to attribute improvements in job performance to training. 

In terms of social networks, those with smaller networks - more specifically, those who had zero 
contacts at baseline - reported more improvement in two areas, completing tasks more accurately and 
with fewer errors, and managing time more effectively. However, for the majority of outcomes, there 
was no link between improvement in job performance and network size. 

In terms of age, older workers reported more improvement than prime-aged workers [35-44) in time 
management, while younger workers reported more improvement than prime-aged workers in 
two areas, related to both time manage and efficiency in completing tasks. However, for the majority of 
outcomes, there was no link between improvement in job performance and age. 

7.6 Mediating Factors that infiuence Changes in Performance 

1) Psychological Capital 

Psychological capital appears to have a more complex and mixed influence on performance. Improvements 
in joh performance were more likely to he reported hy those who had higher levels of motivation and 
engagement at work at baseline, hut also hy those who had low levels of self-efficacy and self-esteem. 

Those with a lower level of maladaptive cognitive issues at work [i.e. a higher level of motivation and 
engagement) reported more improvement in their ability to: 

■ complete tasks more accurately, with fewer errors 

■ be more organized and able to plan their time more effectively 

■ understand and use workplace documents more easily, such as workplace manuals or reports 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order 

■ encourage customers to buy the company's products and services 

Those with a lower level of self-efficacy reported more improvement in their ability to: 

■ understand and use workplace documents more easily, such as workplace manuals or reports 
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■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ communicate with customers, such as describing features and benefits of company products 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order 

Those with low self-esteem reported more improvement in their ability to: 

■ use computers and work-related software programs more effectively 

2) Attitudes towards Learning 

Improvements in job performance were more likely to be reported by those with positive attitudes towards 
learning at baseline 

A broader range of improvements was seen among those who had positive attitudes about learning in 
general at baseline [i.e. those who agree with statements such as "You are more likely to get a better job 
if you do some learning" and disagree that "Getting qualifications requires too much effort"), compared 
to those with similar characteristics but less positive attitudes about learning [see below for details). 

Those who had more positive attitudes towards learning in general reported more improvement in 
their ability to: 

■ be more organized and able to plan their time more effectively 

■ use computers and work-related software programs more effectively 

■ use workplace equipment more effectively, such as tools for my job or office machines 

■ communicate and work more effectively with co-workers, such as completing different job tasks in 
teams 

In addition, those with more positive attitudes towards learning, as well as those who had reported a 
greater number of training goals at baseline, were more likely to attribute improvements in job 
performance to training. 

3) Participant Well-being 

Improvements in job performance were more likely to be reported by those wbo reported at baseline that 
their jobs allowed them the flexibility to fit work in around family life 

In 7 of the 11 areas of job performance, workers who benefited from employer-provided facilities and 
flexibility to fit work in around family life reported greater improvement than their counterparts with 
otherwise identical characteristics but lower levels of work-family balance [see below for details). 
There was no significant difference between groups in the other 4 areas. 
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Those who enjoyed greater degrees of employer-provided facilities and flexibility to fit work in around 
family life reported more improvement in their ability to: 

■ be faster at getting things done 

■ be more organized and able to plan their time more effectively 

■ understand and use workplace documents more easily, such as workplace manuals or reports 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ communicate with customers, such as describing features and benefits of the company's products 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order 

■ encourage customers to buy the company's products and services 

They were also more likely to attribute improvements in job performance to training. 

These results support the idea that workers whose jobs allow them to better balance the everyday 
demands of both work and family life may be in a better position to benefit from training. 

7.7 Implementation Factors that influence Changes in Job Performance 

1) Training Hours 

Improvements in job performance were more likely to be reported by workers in firms that delivered more 
hours of training 

Multivariate analysis revealed that firms offering a greater number of training hours had higher 
worker-reported rates of improvement in 6 of the 11 job performance areas compared to firms with 
otherwise identical characteristics but a lower number of training hours [see below for details). There 
were no significant differences between firms with higher and lower numbers of training hours in the 
other 5 areas. 

However, as with the association between training hours and gains in other participant outcomes, this 
result is somewhat confounded by the fact that all the firms delivering 30 or more hours of training 
were based in Nova Scotia, so it is not entirely clear to what extent the result represents a province 
effect rather than a pure training dosage effect. 

Workers in firms that received a higher dosage of training reported more improvement in their 
ability to: 

■ understand and use workplace documents more easily, such as workplace manuals or reports 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ use workplace equipment more effectively, such as tools for my job or office machines 
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■ communicate with customers, such as describing features and benefits of the company's products 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order 

■ encourage customers to buy the company’s products and services 

Those who received 30 or more hours of training were also more likely than those who received fewer 
than 10 hours to attribute improvements in job performance to training. 

2) Training alignment with business needs 

Improvements in job performance were more likely to be reported by workers in firms in wbicb training was 
relatively well aligned with business needs 

Multivariate analysis revealed that firms with at least a moderate degree of alignment between training 
and business needs had higher worker-reported rates of improvement in 8 of the 11 job performance 
areas compared to firms with otherwise identical characteristics but a lower degree of training- 
business alignment. There was no significant difference between firms with higher and lower degrees 
of alignment in the other 3 areas. 

Workers in firms with higher degrees of business-training alignment reported more improvement 
in their ability to: 

■ be faster at getting things done [well aligned > poorly aligned) 

■ be more organized and able to plan their time more effectively 

■ understand and use workplace documents more easily, such as workplace manuals or reports [well 
and moderately aligned > poorly aligned) 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace [well aligned > moderately aligned > poorly aligned) 

■ use workplace equipment more effectively, such as tools for my job or office machines [well and 
moderately aligned > poorly aligned) 

■ communicate and work more effectively with co-workers, such as completing different job tasks in 
teams [moderately aligned > poorly aligned) 

■ communicate with customers, such as describing features and benefits of the company’s products 
[moderately aligned > poorly aligned) 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order [moderately aligned > poorly aligned) 

■ encourage customers to buy the company’s products and services [moderately aligned > poorly 
aligned) 
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Those from firms in which training was well aligned with business needs were also more likely than 
those in firms with poor training-business alignment to attribute improvements in job performance to 
training. 

7.8 Firm Characteristics that infiuence Changes in Job Performance 

1) Pre-existing training resources 

Improvements in job performance were more likely to be reported by workers in firms that already bad at 
least some pre-existing training resources available at baseline. 

The analysis revealed that firms with at least a moderate amount of pre-existing training resources had 
higher worker-reported rates of improvement in 10 of the 11 job performance areas compared to firms 
with otherwise identical characteristics but lower amounts of pre-existing training resources (see 
below for details). 

Workers in firms with higher amounts of pre-existing training resources reported more 
improvement in their ability to: 

■ be faster at getting things done [moderate resources > few resources) 

■ complete tasks more accurately, with fewer errors [high & moderate resources > few resources) 

■ be more organized and able to plan their time more effectively [high & moderate resources > few 
resources) 

■ understand and use workplace documents more easily, such as workplace manuals or reports 
[moderate resources > few resources) 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace [moderate resources > few resources) 

■ use computers and work-related software programs more effectively [high & moderate resources > 
few resources) 

■ use workplace equipment more effectively, such as tools for my job or office machines [moderate 
resources > few resources) 

■ communicate with customers, such as describing features and benefits of the company’s products 
[high & moderate resources > few resources) 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order [high & moderate resources > few resources) 

■ encourage customers to buy the company’s products and services [high & moderate resources > 
few resources) 
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2) Firm Size 

Improvements in job performance were more likely to be reported by workers in large firms 

Multivariate analysis revealed that large [> 500 employees) firms had higher worker-reported rates of 
improvement in 9 of the 11 job performance areas compared to medium-sized [100-499 employees) 
and/or small [less than 100 employees) firms with otherwise identical characteristics [see below for 
details). 

Workers in firms with higher amounts of pre-existing training resources reported more 
improvement in their ability to: 

■ be faster at getting things done [large firms > small firms) 

■ complete tasks more accurately, with fewer errors [large firms > small & medium-sized firms) 

■ be more organized and able to plan their time more effectively [large firms > small & medium-sized 
firms) 

■ understand and use workplace documents more easily, such as workplace manuals or reports 
[large firms > small & medium-sized firms) 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace [large firms > small & medium-sized firms) 

■ use workplace equipment more effectively, such as tools for my job or office machines [large firms 
> small & medium-sized firms) 

■ communicate and work more effectively with co-workers, such as completing different job tasks in 
teams [large firms > medium-sized firms) 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order [large firms > small & medium-sized firms) 

■ encourage customers to buy the company's products and services [large firms > small & medium- 
sized firms) 

Larger firms may have better supporting infrastructure - including more varied and specialized human 
resources functions - to help workers consolidate the gains they made during training and translate 
them into improved job performance. 

3) Sector 

Improvements in job performance were more likely to be reported by workers in the services sector 

Multivariate analysis revealed that firms in the services sector had higher worker-reported rates of 
improvement in 8 of the 11 job performance areas compared to firms in the manufacturing sector with 
otherwise identical characteristics, which had higher reported rates of improvement in only 1 of the 
11 performance areas [see below for details). 
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Workers in firms in the services sector reported more improvement in their ability to: 

■ be more organized and able to plan their time more effectively 

■ understand and use workplace documents more easily, such as workplace manuals or reports 

■ be better prepared for emergencies and following safety procedures, such as handling an injury or 
fire in the workplace 

■ use workplace equipment more effectively, such as tools for my job or office machines 

■ communicate and work more effectively with co-workers, such as completing different job tasks in 
teams 

■ communicate with customers, such as describing features and benefits of the company's products 

■ respond more effectively to customers' questions or concerns, such as solving problems with an 
order 

■ encourage customers to buy the company's products and services 

Workers in firms in the manufacturing sector reported more improvement in their ability to: 

■ use computers and work-related software programs more effectively 
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Chapter 8; Business Contrasts; Understanding Site-ievei 
Differences 


The methodology used in the Measures of Success project for evaluating longer-term outcomes of the 
Essential Skills training interventions is based on a theory-driven multi-site case study. Specifically, it 
used a Theory-of-Change approach that emphasizes the construction of a rich program logic model that 
outlines all the underlying assumptions upon which a program is expected to lead to a specific result. It 
clarifies how the process will unfold and places attention on the intermediate changes that need to 
occur for long-term outcomes to be reached. Importantly, a theory of change is most effective when it is 
developed through an iterative process that begins with a review and consolidation of prior knowledge, 
followed by a consultation with key stakeholders who have unique knowledge about the possible 
effects of a program and its underlying assumptions. 

Chapter 2 presented the theory-based program logic model for the Measures of Success project that 
was developed through this consultative approach. The power of this framework is illustrated when the 
analysis of the collected data validates the underlying assumptions that link the intervention to the 
longer-term outcomes. In the absence of an experimental design, or quasi-experimental study, this 
approach provides a practical alternative that can provide some credible evidence linking Essential 
Skills training to longer-term outcomes. Pre- and post measures of change not grounded in a richly 
specified theory-driven logic model are subject to bias and lack credibility. 

This chapter reviews the results of the explanatory analysis presented earlier, highlighting the areas 
where results were consistent with theory and expectations regarding the factors that influence 
outcomes of training. It illustrates how these results can help us understand site-level differences in 
outcomes and validate their link to training. 

8.0 Explanatory Factors and Expectations 

The results of the explanatory analysis have validated many of the expectations of stakeholders in this 
project and the consolidated knowledge inherent in the Theory of Change about how workplace 
training interventions can lead to positive outcomes. 

Results have confirmed the relevance of the implementation process: 

■ Business Alignment: A broader range of improvements in job performance was reported by 
trainees from firms where there was a higher degree of alignment between training and business 
needs. 

■ Dosage of training: Participant gains from baseline to 6-months - including post-training skill gains, 
skill practices, and improvements in job performance - were observed among those who received 
more hours of training. 

■ Location: Participant gains from baseline to 6-months were especially likely to be reported by 
workers from firms that delivered the training onsite. 
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Results have confirmed the relevance of some firm characteristics and contextual variables: 

■ Performance Incentives: A broader range of improvements in participant outcomes - including 
post-training skill gains and skill practices - was observed among trainees from firms that offered 
performance incentives. 

■ Flexibility, Employer Supports: A broader range of improvements in job performance was reported 
by trainees who at baseline had had more employer-provided facilities and flexibility that allowed 
them to balance work with family life. 

■ Pre-existing training resources: A broader range of improvements in job performance, as well as 
better business outcomes, was reported by trainees and sponsors from firms that already had some 
pre-existing training resources at baseline. 

■ Size and Sector: A broader range of improvements in job performance was reported by trainees 
from large (500+ employees) firms and from trainees in the service sector than in manufacturing. 

Results have confirmed the relevance of a number of learner characteristics and mediating conditions: 

■ Participant attitudes and goals for training: A broader range of improvements in participant 
outcomes - including post-training skill gains and literacy practices - was observed among those 
who had more positive attitudes and more specific goals about how they would use the training 
they were about to receive. 

■ Participant social capital: While gains were experienced by those with both high and low levels of 
social capital, those with larger networks prior to training experienced greater gains in confidence 
in literacy skills as well as improvements in work-family balance. 

■ Participant psychological capital: These characteristics appear to have a complex and mixed 
influence on performance, where those with both high and low levels are more likely to experience 
gains. Higher gains in some participant outcomes were also realized by those with greater levels of 
maladaptive cognitions. 

8.1 Learner Characteristics, Firm Context, and the Training Process 

Learner characteristics, business context, and the training process were reviewed in Part I of this 
report. Comprehensive site-level differences in these baseline factors were presented in an earlier 
unpublished project report (SRDC, 2012). While results can be analyzed for a single site, they are more 
meaningful when sites are grouped based on expectations related to the context and implementation 
process. This approach is particularly useful in research or evaluation contexts where multiple sites 
exist but the sample sizes are too small to support any significant quantitative analysis. Specifically, 
businesses can be treated as individual cases or grouped according to the baseline conditions and 
whether they either support or hinder positive training outcomes based on expectations [many of 
which have now been validated). 

In many respects, about half of the sample [NSl, NS4, NS5, NS6, MB2, MB4, MBll, MB14, and MB15) 
shared a pattern of indicators suggesting that many conditions were better than average and could 
support positive training outcomes. Overall, they had fewer differences in contextual variables in terms 
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of both participant and business characteristics. Most had a high degree of business alignment and only 
a minority had average or moderate alignment. All had multiple better-than-average mediating factors 
that can support and encourage engagement with training activities, additional use of literacy skills, 
and further learning activities including three or more of the following: strong workplace support; high 
participant motivation; higher than average social capital; and more frequent literacy practices. 

While participants in some of these businesses may have been less likely to realize change because they 
were already at a high level (e.g., psychological capital), baseline levels for some of their intermediate 
outcomes and most of their longer term outcomes were not unusually high. In particular, there was 
room for improvement in several businesses in participant skills, social capital, literacy practices, and 
notably workplace performance measures. Few of their longer-term outcomes were at high levels 
including perceived health and wellbeing and intangible business outcomes. 

A second group of businesses representing about a quarter of the sample also shared a pattern of 
indicators, which suggested mixed conditions that may have or may not have supported positive 
training outcomes. To begin with, they were more distinctive in terms of their participant and business 
characteristics, which may have given rise to differences in future training outcomes for a number of 
reasons. Aligning the training with business needs appears to have been more challenging to achieve 
for most of these businesses. With respect to mediating conditions, there was also a mixed story, with 
some supportive and some potentially inhibiting factors. At the same time, many of the longer-term 
outcomes were well below average levels for a few of these businesses. 

So while the conditions for this group of businesses are somewhat mixed in terms of enabling or 
inhibiting positive training outcomes, participants had much to gain given their lower existing baseline 
levels. As such, improvements in some of the intermediate outcomes were observed in spite of these 
challenges, even in some of the longer-term outcomes where performance was already low. 

A final group of businesses, which made up the remaining quarter of the sample, shared a somewhat 
similar pattern of indicators that suggested a significantly more challenging environment for observing 
positive outcomes of training. They were also fairly distinctive in terms of their participant and 
business characteristics, making customization more vital. Training-business alignment was also more 
challenging to achieve in several of these businesses. With respect to mediating conditions, there were 
no businesses in this group that were above average on a majority of indicators. Longer-term outcomes 
were also mostly below average. While this pattern by no means implies that positive training 
outcomes cannot be achieved, it is likely that these businesses faced additional challenges in achieving 
those outcomes. 

8.2 Business Contrasts 

While it is beyond the scope of this report to present a full set of business level results [confidential site 
level reports were provided to each participating business), this section highlights post-training results 
for few selected businesses based on their pre-training conditions. A selection of outcomes for 
businesses with above average baseline conditions is contrasted with the average outcomes of all firms 
where differences are statistically significant. This demonstrates not only how baseline context and 
implementation process can help explain future outcomes of training - but it also illustrates how 
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transformative training can be when conditions are right. Results for selected businesses where 
baseline conditions were more challenging demonstrate more modest post-training results. 

1) Above Average Conditions: Post-Training Highlights for NS4 

Post-training results for NS4 revealed a range of positive participant outcomes where changes were 
significantly greater than the averages of all other firms. The focus here is on gains in literacy skills and 
practices that were aligned with the goals of the employer. Greater gains were seen in math skills 
[Figure 8.1}, technical skills [see figure 8.2) needed to do their main job well, and frequency of literacy 
practice [see figure 8.3). 

Figure 8.1 Gains in math skiiis: Other Firms vs. NS4 
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In other firms, a high percentage of employees - 81 percent - agreed or strongly agreed that they had 
the math skills needed to do their job well even before training. 6-months after training, this had 
increased slightly to 85 percent. At NS4, on the other hand, before training 0 percent of learners agreed 
that they had the math skills needed to do their job well. However, 6-months after training, this had 
increased to 80 percent. 

Figure 8.2 Gains in technicai skilis: Other Firms vs. NS4 
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In this example, the percentage of employees in other firms who agreed or strongly agreed that they 
had the technical skills needed to do their job well increased from 79 percent before training to 86 
percent after training, this increase was much larger for employees in NS4- from only 40 percent before 
training to 100 percent after training. 
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In addition, 6-months post-training, participants in NS4 reported greater gains than participants from 
other participating firms in the frequency of doing math as part of their daily activities [see figure 8.2 
above). In other firms, the percentage of learners reporting that they did math at least a few times per 
week held steady at just over 50 percent before and after training. However, the increase was much 
larger among learners in NS4, from only 40 percent who did math at least a few times per week before 
training to 100 percent 6-months after training. The finding that learners in NS4 both improved their 
math skills at work and increased their frequency of using math at home shows that the influence of 
training may extend beyond the workplace. Applying newly acquired skills to multiple domains is likely 
to give rise to greater opportunities for skill practice and development, which may in turn have further 
positive consequences in both the workplace and in everyday life. 

Figure 8.3 Gains in frequency of using math: Other Firms vs. NS4 
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2) Above Average Conditions: Post-Training Highlights for MB4 

Post-training results for MB4 revealed a range of positive participant outcomes where changes were 
significantly greater than the averages of all other firms. The focus here is on those gains observed in 
literacy skills and practices, which were aligned with the goals of the employer. 

Six months after training, MB4 participants reported greater gains, when compared with participants 
from all other firms participating in the project, in confidence in their ability to do math [see figure 8.4), 
in reporting being good with numbers/calculations [see figure 8.5), and in confidence in their ability to 
read non-fiction [see figure 8.6), and write notes, letters, and emails, [see figure 8.7). 

Figure 8.4 Gains in confidence in abiiity to do math: Other firms vs. MB4 
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In other participating firms, the percentage of participants that reported being very confident in their 
ability to do math held steady at just over 60 percent before and after training. However, MB4 learners 
reported a large increase, from only 20 percent who reported being very confident in their math 
abilities before training to 80 percent after training. 

Figure 8.5 Gains in reporting being good with numbers and caicuiations: Other firms vs. MB4 


other firm employees 


70% 

60% 

50% 

Percentage of 

learners 20% 

10% 

0% 

-10% 



■ Baseline % 

■ 6 mo % 


Responses to "I am good with numbers and calculations" 


MB4 


Percentage of 
learners 


80% 

70% 

60% 

50% 

40% 

30% 

20 % 

10 % 

0 % 




17% 



- Baseline % 
■ 6 mo % 


Strongly Disagree/ Neutral Agree Strongly Agree 

Disagree 


Responses to "I am good with numbers and calculations" 


While in other firms, the percentage of participants that agreed or strongly agreed that they were good 
with numbers and calculations increased from 64 percent before training to 74 percent after training, 
the increase for MB4 participants was much larger - from only 17 percent before training to 84 percent 
after training. 
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Figure 8.6. Greater gains in confidence in abiiity to read non-fiction 
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While in other firms, the percentage of participants that reported being very confident in their ability to 
read non-fiction increased from 50 percent before training to 58 percent after training, the increase for 
MB4 participants was much larger - from only 20 percent before training to 60 percent after training. 

We see a similar pattern for the ability to write notes. While in other firms, the percentage of 
participants that reported being very confident in their ability to write notes, letters or emails 
increased from 41 percent before training to 51 percent after training, the increase for MB4 
participants was much larger - from only 20 percent before training to 60 percent after training. 
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Figure 8.7 Gains in confidence in abiiity to write notes, ietters and emaiis: Other firms vs. MB4 
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PART IV: Recommendations and Conclusions 
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Chapters; Data Quality and Recommendations for Future 
Instruments 


For the Measures of Success project, SRDC developed 8 survey instruments to collect data. To provide 
recommendations for building instruments that can be used in future projects and programs to assess 
training-related gains and link these gains to explanatory factors, we assessed the measures used 
according to four criteria: 

■ Response rate: Did the measure have sufficiently high response rates at baseline and follow-up to 
allow conclusions to be drawn about the target population? 

■ Distributional properties: Did the measure have sufficient variability in responses at baseline to 
allow for possible changes in key outcomes to be captured at follow-up? Pre-post training 
differences are made up of a combination of true change and measurement error. If the distribution 
of responses at baseline is symmetrical, that is distributed fairly evenly across all the possible 
categories of response, then measurement error is likely to be random - i.e. produce losses and 
gains at roughly equal frequency and of roughly equal magnitudes - so that the losses and gains will 
cancel each other out, and the observed pre-post difference will be an indicator of true change. 
However if the distribution is skewed to one end of the responses, then measurement error may 
not be random. For example, high rates of ceiling responding (responding at the highest possible 
category) at baseline leave no room for gains for a large portion of the sample. As a result, it is far 
more likely that measurement error will produce losses than gains, biasing the measure of pre-post 
difference and reducing our ability to detect true change. 

■ Utility: Did the measure capture statistically significant changes in key outcomes, and/or was it a 
key factor in explaining significant changes in other measures? 

■ Conceptual relevance: Is the measure of crucial conceptual relevance in relation to key constructs 
in the research framework? 

We have divided the measures into several key areas below according to the constructs in the research 
framework, and make recommendations in each area. We have occasionally drawn on findings from 
studies other than Measures of Success to support a recommendation. 

9.0 Measures of confidence in workpiace skiiis - iiteracy and technicai 

These measures were intended to assess the extent to which respondents agree that they have the 
reading, writing, math and technical skills they need to do their jobs well [items 46a-d on the baseline 
survey). 

Discussion: All measures met the first criterion, with relatively high response rates [around 75% of the 
original sample gave valid responses at 6-months). However, measures of reading and writing skills did 
not meet the second criterion as they had high incidences of ceiling responding at baseline [49% and 
44%) compared to measures of math and technical skills [36% for both). High ceiling responding likely 
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played a role in the fact that no significant gains were captured in either reading or writing skills; in 
contrast, significant gains were captured in math and technical skills, at both 3-months and 6-months. 

Recommendation: The measures of confidence in workplace math and technical skills should be 
retained. 

If measures of confidence in workplace reading and writing skills are retained, they should be 
reworked to allow for a better distribution of responses. 

9.1 Measures of practice of workplace skills 

These measures were intended to assess the frequency with which participants typically perform 
various specific tasks at work, each of which involve some combination of reading , writing, document 
use, and/or numeracy (items 43a-e, 44a-e, and 45 a-f on the baseline survey). 

Discussion: These items met the first two criteria as they typically had both high response rates 
(around 75%) and, with a couple of exceptions, low rates of ceiling responding, which suggests that 
they are well-suited to capture possible changes from baseline. 

However, the 16 items between them captured only one significant change from baseline at 3-months 
and one at 6-months - both were decreases, in frequency of reading/writing letters, memos, etc. and 
frequency of counting or reading numbers respectively. Because the measures are strictly associated 
with task frequency rather than complexity or efficiency, the relationship between skill gains and 
changes in frequency is unclear. For example, some respondents could have been given new duties 
after training, with added complexity, but no change in frequency of performing a particular type of 
task. 

As a result, these types of measures did not meet the third and fourth criteria of capturing significant 
change and conceptual relevance respectively. 

Recommendation: The 16 measures of workplace skill practice used in the surveys should not be 
retained. 

If measures of workplace practice are desired, we recommend that the measures address elements 
of performance rather than simply frequency - as for example in the retrospective measures of 
changes of the way participants do their jobs, and the extent to which these changes are attributed 
to training (see #3 below). 

9.2 Measures of Improvement In job performance and attribution of 
Improvement to training 

These measures (items 3a-l in the 6-month survey) were intended to assess the extent to which 
learners had over the past six months improved their performance in specific job areas, including 
productivity, organization, technical skills, emergency preparedness, and relations with co-workers and 
customers. A further single measure (item 4 in the 6-month survey) assesses the extent to which these 
various improvements can be attributed to training. 
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Discussion: These measures differ from the other measures described in this section in that they do not 
rely on baseline and follow-up questions to determine pre-post training differences. Instead, 
retrospective questions, asked only in the 6-month follow-up survey, were used to determine pre-post 
improvements. As a result, the second criterion (distributional properties at baseline) is not applicable 
to these measures. The measures do meet the first criterion - i.e. relatively high response rates. 

Nevertheless, their utility and conceptual relevance could be improved by incorporating changes in 
wording and response categories. The battery of job performance questions asked to what extent 
learners agree that they have improved in various parts of their job, making it difficult to infer the 
direction and magnitude of change. For example, disagreeing or strongly disagreeing that there has 
been improvement may indicate either no change or worsening. Agreeing or strongly agreeing may 
indicate either a small or a large improvement. Rewording the questions to ask about changes in job 
performance, with response categories ranging from varying degrees of worsening to no change to 
varying degrees of improvement, would allow for clearer interpretation. 

Similarly, a single question seeks to identify whether training was the primary reason for improvement 
in job performance, by asking learners to estimate how much of the overall pattern of improvement 
they reported was a result of the training - with response categories varying from "none" to "all." How 
learners answer this question likely depends on the extent to which they interpret "training" in a 
narrow sense to mean only the content of the course they took, rather than the longer-term 
consequences that may have arisen as a result of taking the course. 

Asking the question in the form of an implicit counterfactual - e.g. how likely do you think it is that the 
reported improvement would have occurred if you hadn't taken the training - encourages learners to 
take into account the broader influences training may have had on them, for example motivation to 
keep improving that may have been set in motion by the course. 

Recommendation: The 11 measures of change in job performance used in the surveys should be 
retained, but the questions should be reworded to allow clearer assessment of the direction and 
magnitude of change. 

The measure determining the extent to which reported improvements can be attributed to 
training should be retained, but reworded to better incorporate the full range of possible causal 
influences of training. 

9.3 Measures of confidence in skiiis required for everyday tasks 

These measures were intended to assess the extent to which respondents agree that they have the 
reading, writing, math and digital technology skills needed for daily activities (items 29 to 33 on the 
baseline survey). 

Discussion: These measures met the first criterion, with relatively high response rates at around 75%. 
Though most of them have relatively high incidences of ceiling responding at baseline (40% or above), 
this did not prevent them from capturing change, either at 3-months or 6-months. In both cases, five of 
the six measures showed significant gains. 
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Two additional measures are related to confidence; one asked the extent to which respondents feel 
they are good with numbers, the other the extent to which respondents feel they are anxious when 
figuring out amounts (items 43a and 43b respectively on the baseline survey). Both of these items 
showed similar response rates to the item that measures confidence in ability to do math, but had 
better distributional properties - i.e. ceiling response rates that are half of that for the confidence in 
math item (20-24% compared to 57%). The two additional items also demonstrated a capacity to 
capture change; both showed significant gains at 6-months. 

Recommendation: The measures of confidence in ability to read, write, and use a computer should 
be retained. 

To the extent that the measure of confidence in ability to do math is conceptually similar and 
therefore redundant with measures that ask about being good with numbers and being anxious 
when figuring out amounts, we recommend that only the latter two measures be retained because 
of their better distributional properties. 

9.4 Measures of practice of skills required for everyday tasks 

These measures (items 34 to 39 on the baseline survey) were intended to assess the frequency with 
which participants typically do activities such as reading, writing, math, computer use, and 
library/bookstore visits 

Discussion: All these measures met the first criterion with relatively high response rates at around 
75%. Four of the six - reading fiction, reading non-fiction, doing math, and library /bookstore visits - 
also met the second criterion of relatively low rates of ceiling responding at baseline. All four of these 
measures showed significant gains at either 3-months or 6-months or both. 

The other two measures - writing and computer use - showed relatively high rates of ceiling 
responding at baseline at 37% and 49% respectively, and neither showed significant gains at either 
3months or 6-months. 

Recommendation: The measures of frequency of practice of reading fiction and non-fiction, doing 
math, and library/bookstore visits should be retained. 

If measures of frequency of writing and computer use are to be retained, they should be reworked 
to allow for a better distribution of responses -for example by incorporating an element of 
complexity when performing these activities. 

Discussion: Three additional measures (items 43c-e on the baseline survey) relate to practice of 
reading. The first asks the extent to which respondents read only when they have to, the second the 
extent to which respondents agree that reading is a favourite activity, and the third the extent to which 
respondents enjoy talking about what they read. All three of these items showed similar response rates 
to the items that measure frequency of reading fiction and non-fiction, and had similar (low) ceiling 
response rates. However, the three additional items demonstrated a slightly lower capacity to capture 
change; while two of three showed significant gains at 3-months, none of them showed significant gains 
at 6-months. 
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Recommendation: To the extent that the measures of reading only when one has to, reading as a 
favourite activity, and enjoying talking about what one reads are conceptually similar and 
therefore redundant with measures that ask about frequency of reading jiction and non-fiction , 
we recommend retaining only the latter two measure, particularly if reducing the research burden 
is a priority. 

9.5 Measures of learning and training attitudes and goals 

These measures, a four-item training attitudes scale (items 3a-c & 3f on the baseline survey), were 
intended to assess participant motivation to take the training course and do well. The number of 
specific training goals each participant had were also measured (as an eight-item scale, comprising 
items 4_1 to 4_8 on the baseline survey), as was attitudes towards continuous learning in general (as a 
three-item scale comprising items 27a, b & d on the baseline survey). 

Discussion: All three of these scales had relatively high response rates, though two of the three — 
training attitudes and attitudes towards continuous learning — also had high rates of ceiling 
responding. Nevertheless, all three measures showed good utility as explanatory factors, especially as 
predictors of skill development and/or post-training job performance. 

Recommendation: The scales measuring training attitudes and goals, as well as attitude towards 
continuous learning, should be retained in the baseline survey. 

Training attitudes and goals were only asked at baseline, however learning attitudes were also asked at 
3-months and 6-months to measure possible post-training gains in participant receptivity to further 
learning. No significant gains were found; however emerging evidence from other studies shows gains 
in this measure, in conjunction with gains in various measures of psychological capital (see below), 
when training explicitly incorporates elements of professional conduct development as well as skill 
development. 

Recommendation: If improvement in attitude towards continuous learning is an explicit training 
goal, the scale measuring learning attitudes should be retained in follow-up surveys to allow for 
capture of gains in this area. 

9.6 Measures of psychological capital 

These measures were intended to assess participants' non-cognitive/psychological capacities in terms 
of 


■ Self-efficacy, i.e. the generalized ability to solve problems and reach goals in everyday life, with no 
specific focus on work (a ten-item scale of items 61a-j on the baseline survey) 

■ Self esteem in everyday life, with no specific focus on work ( a single item, #62, on the baseline 
survey) 

■ Resilience, i.e. the generalized ability to adapt to change and recover from adversity in everyday 
life, with no specific focus on work (] a two-item scale of items 64 and 65 on the baseline survey) 
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■ Motivation and engagement at work, an eleven-item scale with three distinct subscales, namely: 

a. Job self-efficacy, i.e. work competence and commitment to improvement (a three-item 
subscale of items 63a-c on the baseline survey] 

b. Planning, task management, and perseverance (a three-item subscale of items 63d-f on 
the baseline survey) 

c. Maladaptive cognitions, i.e. thoughts or behaviours that may have a negative impact on 
job performance (a five-item subscale of items 63g-k on the baseline survey] 

Discussion: Of the three measures with no specific focus on work - i.e. self-efficacy, self-esteem, and 
resilience - all had relatively high rates of responding, though self-efficacy and resilience also had high 
rates of ceiling responding. High ceiling responding meant that these measures had a reduced capacity 
to capture post-training change; indeed, no significant gains were found for either of them. 

Self-efficacy had some utility as an explanatory factor, but was not as conceptually relevant to work 
performance as motivation and engagement on one hand, and more specific confidence-based skills 
measures on the other hand. 

Self-esteem had a low rate of ceiling responding, but limited utility as an explanatory factor and no 
post-training gain was observed. However emerging evidence from other studies shows post-training 
gains in self-esteem, in conjunction with gains in other measures of psychological capital, when training 
explicitly incorporates elements of professional conduct development as well as skill development. 

Recommendation: The 1 0-item self-efficacy scale and 2 -item resilience scale should not be 
retained. 

If a measure of generalized psychological capacity, not specifically related to work, is desired, we 
recommend that the single item measure of self-esteem be retained. 

Discussion: In terms of motivation and engagement at work, all three subscales had relatively high 
response rates, though two -job self-efficacy and planning/perseverance - also had high rates of ceiling 
responding which limited their utility in capturing post-training gains. Indeed significant post-training 
gains were only seen in one item Qob anxiety] from the maladaptive cognitions subscale. The 
maladaptive cognitions subscale also showed utility as an explanatory factor, especially as a predictor 
of skill gains at 3-months and job performance at 6-months. 

Despite the lack of post-training gains in job self-efficacy and planning/perseverance in this study, 
emerging evidence from other studies shows significant gains in all three motivation and engagement 
subscales when training explicitly incorporates elements of professional conduct development as well 
as skill development. 

Recommendation: All three subscales of the motivation and engagement at work scale should be 
retained, especially if improvement in motivation and engagement is an explicit training goal. 
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9.7 Measures of social capital, inclusion and cohesion 

Social capital measures included numbers of contacts that can relied upon for various specific kinds of 
help (items 52a-d on the baseline survey] as well as overall network size (item 53 on the baseline 
survey]. Frequency of receiving specific kinds of help (items 58a-d] and network composition (items 
54-57] were also measured. 

Discussion: In general, measures of social capital had lower response rates than most other survey 
items, but good distributional properties with low rates of ceiling responding. Measures of network size 
and composition showed good utility, in capturing post-training change and/or explaining change in 
other measures. Network size and composition are also conceptually relevant as channels through 
which skills can be developed and used. 

Measures of frequency of help received were less useful, since they did not take into account whether 
participants needed help. Frequency of receiving help when needed would be a more relevant concept. 

Recommendation: The measures of network size and composition should be retained. 

If measures of frequency of help received are desired, we recommend that these be reworded to 
measure frequency of help received when needed. 

Discussion: Measures of cohesion include vectors of trust towards both closer and more distant 
network connections (items 59a-f on the baseline survey]. Measures of trust had among the lowest 
response rates of all survey items, and some (especially trust of police officers, co-workers and 
supervisors] had very high rates of ceiling responding. However, some trust measures (neighbours, 
grocery clerks, and strangers] had good utility in that they captured post-training gains and movement 
from closer to more distant vectors of trust. 

Measures of inclusion included frequency of formal and informal volunteering (items 50 and 51 on the 
baseline survey], as well as participation in groups/organizations (items 49a-i on the baseline survey] 
and workplace committees (item 47 on the baseline survey]. These measures had among the lowest 
response rates of all survey items - especially volunteering for which only 40% of the original sample 
responded at both baseline and follow-up - though they showed no tendency towards high rates of 
ceiling responding. 

In general, measures of trust and inclusion, along with social capital, are conceptually useful as 
channels through which skills can be developed and used. 

Recommendation: Only a portion of cohesion and inclusion measures should be retained, namely 
trust of neighbours, grocery clerk, and strangers, as well as participation in groups/organizations 
and workplace committees. 

9.8 Measures of well-being at work and in life 

These measures include life satisfaction (item 60 on the baseline survey], as well as adequate 
employer-provided facilities and flexibility to fit work in around family life (item 48e on the baseline 
survey], flexible working hours /patterns (item 48n on the baseline survey], and satisfaction with 
overall quality of working life (item 48x on the baseline survey]. 
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Discussion: With the exception of life satisfaction which had relatively low response rates, all of these 
measures met all four retention criteria. None had high ceiling responding; all captured significant post- 
training gains at 3- and/or 6months; the three work-related items were also useful as explanatory 
factors; and all were conceptually relevant in terms of being far-reaching outcomes of training 
interventions that support many types of supplementary analysis. 

Recommendation: The measures of life satisfaction, employer-provided facilities and flexibility for 
work/family balance, flexible working hours/patterns, and overall quality of working life should 
be retained. 
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Chapter 10: Concluding Summary 

The Measures of Success project was intended to address three major research questions: 

■ What are the long-term outcomes of workplace Literacy and Essential Skills (LES) initiatives in 
Manitoba and Nova Scotia on the participants and businesses (six months after training]? 

■ What is a valid and reliable evaluation model for assessing longer-term outcomes of workplace 
LES initiatives and what are the appropriate measures to be used? 

■ What do the results tell us about effective and efficient ways to provide workplace LES initiatives 
to maximize positive long-term outcomes? 

What follows is a summary of key results that address each of these core research questions. 

The first set of findings illustrates the kinds of long term outcomes that have been observed in 
Manitoba and Nova Scotia among the 226 participants and 18 firms in the study. It also highlights a 
number of outcomes for which changes were not observed, contrary to expectations. 

The second set of findings highlights the successes of the evaluation model in providing multiple lines 
of evidence which demonstrate that outcomes do in fact relate to the training interventions. It also 
summarizes some of the challenges in data collection and quality, and offers recommendations for 
improving the measures. 

The third set of findings identifies a number of factors that can help explain the pattern of outcomes 
across different contexts, in terms of implementation factors, and participant and firm characteristics. 
This provides some evidence about how workplace LES initiatives can effectively produce positive 
longer-term outcomes. 

10.0 Long Term Outcomes of Workplace LES Initiatives 

1) The evaluation design was effective in measuring longer-term post-training gains across a 
broad spectrum of outcomes including human capital, social capital, well being, and a rich 
set of performance measures for participants and businesses. 

Human capital: There was a wide range of statistically significant improvements in outcomes related to 
human capital from baseline to 6-months post-training, including: 

■ Increased confidence in work-relevant literacy skills such as technical skills and math skills 
trainees needed to do their main job well. 

■ Increased confidence in everyday literacy skills such as reading, writing, math, and computer use. 
There were significant post-training changes in three different math indicators - including 
increased confidence in doing math and being good with figures and calculations, as well as 
reduced anxiety associated with figuring out amounts. 
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■ Increased practice of everyday literacy skills, especially reading. There were significant post- 
training increases in reading fiction and non-fiction, as well as visiting libraries and bookstores. 

Social capital and well-being: In addition to direct gains in skills and practices, there was a variety of 
improvements in several mediating factors that may act indirectly to facilitate further skill 
development and longer-term performance improvement, including 

■ Improved Social capital, such as network size and composition. In addition to gains in network 
size, participant networks also expanded in terms of breadth and types of support they could 
provide. For example, a substantial number of participants went from closed, completely 
interconnected networks, to having less dense networks, which are better for leveraging other 
kinds of resources and supports, including further learning opportunities. 

■ Enhanced Cohesion, such as increased trust. Participants' levels of trust in close connections such 
as neighbours, co-workers and supervisors remained high throughout the follow-up. However, 
post-training, they also broadened their spheres of trust to include more distant connections, as 
exemplified by increased levels of trust in total strangers. Enhanced trust with more distant 
connections may facilitate longer-term development and use of social networks. 

■ Increased well-being, including satisfaction with life and work. While gains in overall life 
satisfaction that were present at 3-months were no longer significant at 6-months, improvements 
in several indicators of job satisfaction were maintained 6-months post-training. This included 
reduction in job anxiety, improvement in the ability to balance the demands of work and family, and 
increased satisfaction with the overall quality of working life. 

Performance and business benefits: Improvements in eleven different job performance indicators 
were reported by participants. At the same time, a majority of firms reported a number of business 
outcomes explicitly linked to training. 

■ Productivity gains included things such as completing tasks more accurately with fewer errors, 
planning time more effectively, and using workplace documents and equipment more effectively. 
Improvements in Interpersonal relations such as better communication with co-workers and 
customers, and ability to respond to customer questions and concerns. Depending on the indicator, 
between 50% and 75% of participants agreed or strongly agreed that they had improved over the 
past 6 months; of those that reported improvement, about three-quarters attributed at least some 
of the improvement to training. 

■ Businesses reported training-related improvements in a variety of areas including both 
tangible and intangible outcomes. At least half of participating employers reported training-related 
improvements in productivity, empioyee-management reiations, iearning cuiture, and costs and 
errors. 
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2) Though there were substantial changes across a broad spectrum of outcomes, there were 
some for which no change was observed, contrary to expectations. 

■ No gains in practice of work-relevant literacy skills. The lack of improvement in these measures 
was likely related to data quality issues, namely that these measures assessed only changes in task 
frequency without taking into account complexity or efficiency. 

■ No gains in psychological capital, including motivation and engagement at work and attitudes 
towards continuous learning. Rather than simply being factors that mediate gains in other skills, 
these measures may represent a unique set of non-cognitive skills in their own right that may need 
to be explicitly targeted by training in order to see improvement. 

■ No gains in indicators of social inclusion, including formal and informal volunteering, and 
participation in groups, organizations, and workplace committees. The lack of improvement in at 
least some of these indicators was likely related to data quality issues, especially very low response 
rates. 

■ No gains in earnings. Six months is likely too short a time frame to capture significant changes in 
earnings. In addition earnings data were compromised by very low response rates. 

10.1 The Evaluation Model and its Measures 

1) The Evaluation Model was reasonably successful at illustrating a link between LES training 
and outcomes by providing multiple lines of rich evidence even though precise impact 
estimates cannot be achieved with this design. 

The comprehensive nature of the evaluation framework and its measures allowed for multiple lines of 
evidence to help demonstrate a link to training. Among other factors, this included an analysis of 
participant characteristics, firm context, mediating conditions, and implementation and training 
delivery. In addition, the inclusion of questions that implicitly aim to establish attribution to training 
proved useful. 

■ Multivariate explanatory analysis of implementation factors such as business alignment and 
training dosage, as well as a range of mediating and contextual factors, reveals multiple links 
between training and improved participant and business outcomes. 

■ Inter-firm comparisons reveal a pattern of post-training outcomes that are largely consistent with 
theory and expectations, such that gains are more likely for those firms that had relatively 
advantageous implementation conditions and with presence of favourable contextual and 
mediating factors. 

■ Participant attribution: A sizable majority (about three-quarters) of participants who reported 
improvement in various job performance indicators attributed at least some of these improvements 
to training. 
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■ Business attribution: More than 60% of improvements reported by firms in various business 
outcomes were attributed by employers to the training initiative. If outcomes that were judged to 
have been "possibly" improved by training are counted, improvements attributed to training rise to 
more than 90% of all improvements. 

2) The study has produced a series of recommendations for improving outcome measurement 
and streamlining instruments in a way that can increase their reliability and better address 
operational challenges of data collection. 

An assessment of data quality in the current study has facilitated a series of improvements to the 
measures and instruments including additions, deletions, and edits to question items to increase their 
reliability. These recommendations were based on a review of several criteria: 

■ Response rate - Did the measure have sufficiently high response rates at baseline and follow-up to 
allow for conclusions to be drawn about the target population? 

■ Distributional properties - Did the measure have sufficient variability in responding at baseline 
to allow for possible changes in key outcomes to be captured at follow-up? 

■ Utility - Did the measure capture statistically significant changes in key outcomes, and/or was it a 
key factor in explaining significant changes in other measures? 

■ Conceptual relevance - Is the measure conceptually relevant in relation to key constructs outlined 
in the research framework? 

In addition, we encourage the development of a few new measures, especially in the area of participant 
attribution of improvement to training. Emerging research is beginning to show that the most effective 
way to ask participants about training impact is to pose the question in the form of an implicit 
counterfactual - e.g. how likely do think it is that the reported improvement would have occurred if you 
hadn't taken the training? This kind of question encourages learners to take into account not only the 
impact of the course itself, but also the broader, longer-term influences training may have had on their 
own motivation and subsequent approaches to learning and problem solving. 

These kinds of implicit counterfactual measures may be crucial to allow for the examination of 
training impacts in the absence of a comparison group. 

10.2 Lessons in Effective Deiivery 

1) Post-training gains were common among a wide range of learners, many that are often 
thought to benefit less from training, including those with lower education, immigrants with 
English as a second language, and older workers. 

■ Lower levels of educational attainment. For the majority of outcomes in which improvements 
were seen in the sample as a whole, participants with high school or less were as likely to show 
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post-training gains or to report improvements in job performance as those with college or 
university credentials. 

■ Immigrants who commonly use languages other than English at home. For the majority of 
outcomes in which improvements were seen in the sample as a whole, immigrants who likely used 
English as a second language [ESL) were as likely as Canadian-born participants to show post- 
training gains. In fact, ESL participants were more likely than others with similar characteristics to 
report improvements in a number of job performance areas. 

■ Older workers [age 45+) were as likely as younger and prime-aged workers to show post- training 
gains or to report improvements in job performance. 

2) Results of the study suggest that a number of important implementation and contextual 
factors can influence the effectiveness of workplace LES training. 

While further research is needed to explore the reasons and confirm the relevance of these findings to 

different populations [i.e. varying outcomes by firm size and sector), many of the findings confirm 

hypotheses and expectations regarding effective training delivery. 

1) Business Alignment: A broader range of improvements in job performance was reported by 
trainees from firms where there was a higher degree of alignment between training and business 
needs. 

2) Dosage of training: a broader range of improvements in participant outcomes - including post- 
training skill gains, skill practices, and improvements in job performance - were observed among 
those who received more hours of training. However, the findings also suggest that positive 
outcomes are also achievable even with smaller dosages as long as other conditions are met 
[alignment, positive attitudes, etc.). 

3) Pre-existing training resources: A broader range of improvements in job performance, as well as 
better business outcomes, was reported by trainees and sponsors from firms that already had some 
pre-existing training resources at baseline. 

4) Participants attitudes and goals for training: A broader range of improvements in participant 
outcomes - including post-training skill gains and literacy practices - was observed among those 
who had more positive attitudes and more specific goals about how they would use the training they 
were about to receive. 

5) Performance Incentives: A broader range of improvements in participant outcomes - including 
post-training skill gains and skill practices - was observed among trainees from firms that offered 
performance incentives. 

6) Flexibility, Work-Life Balance: A broader range of improvements in job performance was 
reported by trainees who at baseline had had more employer-provided facilities and flexibility to 
allow them to balance work with family life. 

7) Size and Sector: A broader range of improvements in job performance was reported by trainees 
from large [500+ employees) firms and from trainees in the service sector than in manufacturing. 


MEETING EXPECTATIONS: Measuring the Impacts of Workplace Essential Skills Training 
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Appendix A.l 

Guidelines for Using the Learner Question Inventory 


The inventory of recommended questions includes: 

• 100 pre-training questions, for possible inclusion in a Baseiine Survey. This is reduced from 186 
questions in the original Measures of Success Baseline Survey, which represents a likely cut in 
response time of about half, from about 40-44 minutes to 20-22 minutes. Cuts were also 
focused on open-ended questions, which generally have a higher response burden and may 
further reduce the overall response times. 

• 115 post-training questions, for possible inclusion in a Foiiow-up Survey. This is reduced from 
178 questions in the original Measures of Success 6 Month Follow-up Survey, which represents 
a cut in response time of about 40-45 per cent, from about 38-42 minutes to 22-24 minutes. 

If further cuts are desired, shorter surveys can be constructed by omitting selections from one or more 
of the subject-matter modules described below, based on evaluation priorities, target groups, response 
burden, and/or cost. The approximate times needed to complete each module are given in the table 
below. 


Module 

Baseline Survey 

Follow Up Survey 


Number of 
questions 

Time 

Number of 
questions 

Time 

A - Literacy/Essential Skills 

13 

3 minutes 

13 

3 minutes 

B - Job Performance 



22 

4 minutes 

C + K - Education & Learning 

7 

1-2 minutes 

6 

1-2 minutes 

D - Psychological Skills 

17 

3 minutes 

17 

3 minutes 

E - Social Networks, Inclusion & 
Cohesion 

21 

4 minutes 

21 

4 minutes 

F + L - Health and Well-Being 

9 

2 minutes 

6 

1-2 minutes 

G - Labour Market Outcomes 

14 

3 minutes 

17 

4 minutes 

H - Training Feedback 



13 

2 minutes 

1 - Training Attitudes and Goals 

6 

1-2 minutes 



J - Demographics & Household 
Characteristics 

13 

3 minutes 



TOTAL 

100 

20-22 

minutes 

115 

22-24 

minutes 


Though each survey is estimated to take 20+ minutes to complete if all the questions are used, it would 
be relatively easy to construct a 15 minute survey. For example, if response burden is a concern, cuts 
can be focused on Modules E and G, each of which features some questions with relatively low response 
rates. In some cases, cuts may be dictated by evaluation priorities - for example, if labour market 


90 



Measures of Success 
Final Report - March 2013 


outcomes are not expected to take place within the time frame of the evaluation, module G can be cut 
or reduced. Cuts may also be based on target groups - for example, if the workplace context is not likely 
to feature significant numbers of immigrants, those with pre-existing health conditions, or those with 
second jobs, cuts can be made to modules J, I, and G respectively. 

Detailed descriptions of each module follow below. 

Modules A and B 

These modules focus on literacy/essential skills and performance outcome measures which can be used 
to evaluate skill and performance gains in areas commonly targeted by training. Both modules feature 
questions that can be used to assess post-training improvements in the absence of objective measures 
of skill gain. Some of these outcomes may often be precursors of longer-term labour market gains (see 
Module H below). 

Module A - Confidence in and Use of Skills focuses on measures of post-training gains in confidence and 
use of literacy/essential skills at home and in the workplace. In the Measures of Success project, the 
questions below were asked at both baseline and 6-month follow-up, allowing us to capture a wide 
range of long-term gains in confidence and use of literacy/essential skills at home and at work. 

Questions 1 to 4: These are measures of confidence in various literacy/essential skills used in the course 
of everyday life. 

Questions 6 to 9: These are measures of literacy/essential skill practice in the course of everyday life. 

Questions 10 to 12: These are measures of confidence in various literacy /essential skills used at work. 
The questions only represent a sample of possible workplace skills; additional skills (e.g. computer skills, 
problem-solving skills, document use skills) may be added as needed. 

Module B - Improvements in Job Performance and Attribution of Improvements to Training focuses on 
measuring improvement in specific job performance areas as well the role training played in that 
improvement. 

Questions 13 to 34: Gathering information on how training may have improved learner job performance 
is challenging in the absence of a counterfactual - that is, a comparison group of similar employees who 
had not participated in the training and whose outcomes could therefore be considered equivalent to 
those learners would have experienced if they hadn't received the training. 

Self-report measures may be viable alternatives if learners are first asked about the degree of post- 
training improvement and then asked how likely the reported improvement would have been if they 
hadn't received the training. This kind of measure is often called an implicit counterfactual, and in the 
absence of a comparison group it may provide the best approximation of outcomes learners would 
have experienced if they hadn't received the training. 
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This module provides examples of these kinds of measures, pairing questions about the magnitude of 
improvement in specific job performance areas with questions asking learners to estimate the likelihood 
that the reported improvement would have taken place without training. Thus, performance areas may 
be compared in terms of which showed the greatest training-attributed improvement. 

The questions provided are based on job performance indicators that were important in the Measures 
of Success study - similar questions for different performance indicators may be developed as needed in 
other training programs/contexts. 

Modules C and D 

These modules focus on education and psychological outcome measures, which can be used to 
evaluate various non-literacy based Essential Skills that may be targeted by training, such as receptivity 
to continuous learning, participation in further education, various dimensions of motivation and 
engagement at work, and future orientation. These kinds of skills are linked to job performance and may 
often be precursors of longer-term labour market outcomes (see Module H below). 

Module C - Participation in Education and Attitudes towards Continuous Learning 

Questions 35 to 37: These questions identify other types of education or training learners might have 
taken since their participation in the training program being evaluated. These other kinds of learning 
participation may be of interest both as post-training outcomes (e.g. did the percentage of learners who 
pursue education or training increase in the aftermath of the targeted training program?) and as 
possible predictors of other post-training outcomes (e.g. are post-training outcomes better for those 
who participated in other forms of learning in addition to the targeted training program?). 

Question 38: This measure assesses attitudes towards learning in general. 

The three items in question 38 (38a, 38b, and 38c) comprise a learning attitudes scale, and provide a 
measure of learners' receptivity to continuous learning. In the Measures of Success study, this measure 
was a useful predictor of post-training job performance, and so may serve to identify learners who are 
likely to benefit from the training. In other studies, the scale has proven to be a useful outcome 
measure, as receptivity to continuous learning has shown significant improvement in training contexts 
that target professional conduct and career development. 

Module D - Psychological Skills 

Questions 39 to 41 comprise measures of psychological skills - i.e. personality traits and motivation - 
that the research literature shows are linked with workplace performance and socioeconomic outcomes 
in general. These kinds of traits often determine how effectively learners apply their literacy/essential 
skills and other educational attainments, and can be thought of as skills in their own right. 

Question 39 is a single-item measure of self-esteem. 
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Question 40 combines several items into a scale to assess various dimensions of employee motivation 
and engagement at work, including the positive and negative ways people perceive their jobs, as well as 
the positive and negative ways they behave at work. In more formal terms, the scale consists of : i) 
adaptive cognitive dimensions such as self-efficacy, mastery orientation, and valuing (40a, 40b, and 40c), 
ii) adaptive behavioural dimensions such as planning, task management and perseverance (40d, 40e, 
40f), iii) maladaptive cognitive dimensions such as anxiety, fear of failure, and perceived lack of control 
(40g, 40h, 40i), and iv) maladaptive behavioural dimensions such as self-handicapping and 
disengagement (40j and 40k). 

Question 41 combines several items into a scale to assess future orientation. Research indicates that 
future-focused learners are willing to defer immediate benefits in order to receive larger, long-term 
benefits down the road, traits that are also associated with conscientiousness, perseverance, problem- 
solving on the job, and receptivity to further education. 

There were few post-training gains in psychological indicators used by the Measures of Success project 
(i.e. self-esteem and motivation and engagement at work, questions 39 and 40 above). However, 
emerging research from other projects shows significant gains in these indicators as well as future 
orientation (question 41 above) when training programs explicitly incorporate elements of professionai 
conduct and career-oriented thinking as well as literacy/essential skill development into their curricula. 

Although the vast majority of measures in the question inventory are free to use, use of the motivation 
and engagement scale (question 40) requires a nominal licensing fee - e.g. for the Measures of Success 
Study, CDN$100 for a 2-year license. Alternatively, new items may be developed to capture similar 
skills/concepts. 

Modules E and F 

These modules focus on additionai outcome measures which can be used to evaluate broader post- 
training gains in areas such as social network size and composition, social inclusion and cohesion, as well 
as health and well-being, including satisfaction with work and life in general. These kinds of indicators 
may be important for two reasons: 

i) They may represent important mediating factors that may in some contexts constrain the 
effectiveness of training. For example, poor health, small social networks, and low levels of social 
participation, trust, or work/life satisfaction may create inhibiting conditions for iearning transfer, i.e. 
limited opportunities to use and apply newly acquired skills. 

ii) On the other hand, if training leads to improvements in one or more of these factors, it may create 
faciiitating conditions for iearning transfer in terms of new opportunities and increased motivation for 
learners to practice and further develop newly acquired skills both in the workplace and in everyday life. 

Moduie E - Sociai Networks, Inciusion, and Cohesion 
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Questions 42 and 43 are indicators of social inclusion - that is participation in groups or organizations at 
home and at work respectively. 

Question 44 and 45 include several different indicators of social network size; for example, size of 
networks that enable access to various specific kinds of resources (help with household activities, 
emotional support, specialized advice, help with job or career) as well as overall network size. 

Questions 46, 47, and 48 are measures of network composition, which incorporate both 
interconnectedness (greater depth of support) and diversity (greater breadth of support). 

Question 49 focuses on social cohesion by asking about trust towards both closer and more distant 
network connections. 

The Measures of Success study found long-term post-training gains in overall network size and network 
diversity. Learners developed not only larger social networks that they could rely upon for various kinds 
of help and support, but many of them also diversified their networks to include new types of work 
and/or non-work contacts they had not had before. 

Learners also showed post-training gains in trust, an important indicator of social cohesion which can 
facilitate interaction and networking, and thus expand and diversify social networks further. The 
recommended indicator for trust is a commonly used measure based on participant perceptions of the 
likelihood that a lost wallet will be returned if found by different members of a community - which may 
include work colleagues, neighbours, civil servants, and/or total strangers. 

In general, newly acquired skills - for example communication and problem-solving - may allow learners 
to expand and diversify their networks as well as their sense of trust, and this in turn may create 
facilitating conditions for further learning as described above 

Module F - Health and Well-Being Outcomes 

Questions 50 and 51 provide indicators of both general and mental health. 

Question 52 provides several indicators of satisfaction or contentment with job and career, including 
home-work interface (52a and 52b) as well as overall quality of working life (52c). 

All three work satisfaction indicators showed significant post-training gains in the Measures of Success 
study - however, the first two (52a & b) are part of a proprietary 23-item Work-related Quality of Life 
Scale. Use of the scale requires a licensing fee of close to $500, though it is unclear what the cost of 
using only a small part of it would be. If necessary, new items may be developed to capture similar 
concepts. In general, improvements in work satisfaction may lead to not only a variety of positive 
business outcomes, but also increased learner motivation to practice and further develop newly 
acquired skills. 

Question 53 is a commonly used indicator of satisfaction with life as a whole. 
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Module G - Labour Market Outcomes 

This module focuses on longer-term labour market outcomes, including wages and salaries, career 
advancement, and job stability. There were no significant post-training labour market gains in any of 
these outcomes observed in Measures of Success. However, labour market outcomes typically take 
longer to appear than most of the other outcomes above, and so may require a longer period of post- 
training evaluation than the 6 months used for Measures of Success. 

Questions 57 to 59 cover earnings, as well hours worked per week and weeks per month to capture 
earnings gains that may result from increased hours as well as those resulting from wage increases. 

Questions that ask participants to give their exact earnings are often sensitive and may have low 
response rates, so we also recommend asking about post-training increases in wages/earnings 
(Question 69). As described more fully for the job performance measures in Module B (above), this 
question is accompanied by an Implicit counterfactual measure, asking participants to estimate the 
likelihood that the reported earnings increase would have taken place without training (Question 70). In 
the absence of a comparison group, this may represent the best alternative way to estimate the impact 
of training on earnings. 

Career advancement may be indicated by changes in the job title over the employee's tenure at the firm 
(Questions 54 to 57), as well as by asking directly about post-training promotions (Question 71). An 
accompanying Implicit counterfactual measure, asking participants to estimate the likelihood that the 
reported promotion would have taken place without training (Question 72), may represent an 
alternative way to estimate the impact of training on career advancement in the absence of a 
comparison group. 

Job stability may be measured by a question on reductions in hours worked (Question 59), as well as 
questions on job permanence and work stoppages experienced (Questions 61 to 63). 

A series of questions about a second Job are also included (Questions 64 to 68). Though only about 13% 
of the Measures of Success sample held a second job, in other employment contexts where part-time or 
temporary jobs are more frequent, questions on second jobs may serve as useful indicators of career 
advancement or job stability. For example, in some cases post-training gains associated with the main 
job may allow participants to drop their second job. In other cases, post-training gains may allow 
learners to get a better second job. 

Module H - Training Feedback, Course Quality and Relevance 

This module focuses on short-term course-related outcomes such as learner satisfaction and the extent 
to which the training was relevant to the job. 

Questions 73 to 76: These questions are best asked soon after the training has been completed, as part 
of a short-term follow-up survey. They would provide important feedback to instructors, employers and 
other stakeholders on how learners rate the training. 
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If desired, a few of these questions can also be asked retrospectively as part of a longer-term follow-up 
survey, for example to determine to what extent learners still feel the training is relevant to their jobs 
(question 73g), the extent to which they have been able to apply what they learned to their job (73h), 
the extent to which there are opportunities to practice their new skills on the job (73i), or the extent to 
which they would recommend the course to co-workers (73j). 

In addition, the overall course rating (question 76) can be re-asked in the longer-term as well, to 
determine whether learners who have had the opportunity to try out some of the things they learned 
value the course more or less than they did shortly after the training. 

The questions in this module were the ones used in the Measures of Success study; they are not 
necessarily intended to replace similar questions that may be already in use as part of existing training 
evaluations. 

Modules I, J, K and L 

These modules focus on learner characteristics at baseline -such as training motivation, age, 
household composition, economic hardship, language barriers, educational attainment, and disability - 
that may act as mediating factors either constraining or supporting possible post-training gains. These 
kinds of measures would allow users to evaluate the extent to which training is reaching a range of 
possible targeted sub-populations. 

Module I - Training Attitudes and Goals 

Question 77 : Four of the items in this question (77a, 77b, 77c, and lie) comprise a training attitudes 
scale, providing a measure of the learners' motivations with regard to the training they are about to 
take. In the Measures of Success study, this measure was a useful predictor of several positive post- 
training outcomes. Thus, it may serve to identify learners who are likely to benefit from the training. 

The fifth item (77d) identifies whether learners have the support of their supervisor. In the Measures of 
Success study, the vast majority of learners agreed that they had the support of their supervisor. 
However, in cases where supervisor support varies to a greater extent, it may be an important mediator 
of training success. 

Question 78 : This question: a) identifies whether the training was voluntary or a job requirement, and b) 
enumerates the number of specific goals each learner had as they entered the training. In the Measures 
of Success study, number of training goals was a useful predictor of several positive post-training 
outcomes. Along with training attitudes (above), it may serve to identify learners who are likely to 
benefit from the training. 

The specific goals listed in the question are not intended to be definitive; goals different from those 
listed may be included depending on the training context and delivery. 

If 'job requirement' is checked as a reason for taking training, it is typically not counted in the 
enumeration of training goals, but rather used as an indicator of whether the learner perceived the 
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training as voluntary or compulsory. In the Measures of Success study, there was no clear relation 
between post-training outcomes and whether training was perceived as voluntary or compulsory. 
However, in other contexts, it may be an important mediator of training success. 

Module J - Demographics & Household Characteristics 

Questions 81 to 85: Questions on marital status, household composition and household income serve to 
identify learners who may be economically constrained (e.g. single parents, lower income household, 
presence of pre-school aged children) and who thus may be in less of a position to benefit from training, 
particularly if it is offsite or outside normal working hours. These measures allow programs to evaluate 
whether training is reaching learners with these characteristics. 

Questions 86 to 88: These questions identify learners who are immigrants (recent or established) or 
Aboriginal persons, which may be important for programs targeting these often multi-barriered groups. 

Questions 89 to 91: In the absence of an objective measure, responses to these questions can be 
combined to generate approximate groupings based on proficiency in one of the official languages. For 
example, immigrants may be classified according to four levels of probable proficiency in English: 1) 
those whose mother tongue is English and who speak English most often at home; 2) those with a non- 
official mother tongue, but who speak English most often at home; 3) those with a non-official mother 
tongue, who speak a non-official language most often at home, but who also speak English regularly at 
home; and 4) those with a non-official mother tongue, who speak a non-official language most often at 
home, and do not speak English regularly at home. 

These kinds of groupings may be especially important for programs seeking to target a largely immigrant 
workforce, and to evaluate whether training is reaching those who may be less proficient in English. 

Module K - Previous Education and Training Experience 

Questions 92 & 94: These questions - along with Question 35 from Module C, asked at baseline - 
identify learners' level of educational attainment, as well as recent pre-training experience with formal 
education or other kinds of training. They are useful in helping to determine the extent to which 
previous participation in formal education may be linked with training outcomes. For example, the 
Measures of Success evaluation found that those with lower levels of educational attainment were no 
less likely that those with college or university degrees to benefit from the training. 

Questions 93: This question measures the extent to which learners' previous educational experiences 
were positive or negative. 

Module L - Health/Disability Status 

Questions 95 to 97 are measures of the extent to which physical or mental conditions may impede 
activities at work and in general. These indicators may represent important mediating factors that may 
in some contexts constrain the effectiveness of training. They may be especially important for programs 
seeking to target a workforce for whom disabilities may represent significant barriers to labour market 
advancement. 
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Appendix A.2 

Question Inventory for Learners 


A comprehensive inventory of recommended questions, to allow stakeholders to construct measurement instruments of varying lengths 
depending on the evaluation priorities of the users, is presented below. 

The questions are organized conceptually according to subject-matter modules, not necessarily according to the order in which they should be 
asked. 

Beside each question is an indicator as to when it should be asked: 

B = BASELINE ONLY; pre-training only 

F = FOLLOW-UP ONLY; post-training only 

BF = BASELINE AND FOLLOW-UP; both pre- and post-training 

For certain questions, there are skip indicators for some responses. An S indicates that an instruction to skip a subsequent question(s) should be 
added (e.g. "Skip to question "), with the number following the S indicating the number of questions that should be skipped. 

An accompanying Guide with descriptions of modules and key measures in each module is attached to help stakeholders prioritize and make 
decisions about which questions to use. 
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A) Confidence in and Use of Skills 

Suggested heading : "Information about activities at home and at work" 


Question 

Response 




The next questions are about how confident you feel in doing 
various activities 

Not At All 
Confident 

Not Too 
Confident 

Somewhat 

Confident 

Very Confident 

1. BF How confident are you in your ability to read fiction 
(stories, novels)? 

1 

2 

3 

4 

2. BF How confident are you in your ability to read non-fiction? 
(e.g., history, religious, science, self help books; reference 
materials; news sections of newspapers and current affairs 
websites) 

1 

2 

3 

4 

3. BF How confident are you in your ability to write notes, letters 
or email? 

1 

2 

3 

4 

4. BF How confident are you in your ability to use a home 
computer? 

1 

2 

3 

4 


Question 

Response 





5. Please indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

disagree 

Disagree 

Neutral 

Agree 

Strongly 

agree 

a) BF 1 am good with numbers and calculations. 

1 

2 

3 

4 

5 

b) BF 1 feel anxious when figuring out such amounts as 
discounts, sales tax or tips 

1 

2 

3 

4 

5 
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Question 

Response 






The next questions are about how often you do 
various activities 

Never 

Rarely 

Less than 
once a week 

Once a week 

A few times a 
week 

Every day 

6. BF How often do you do math (such as for bills, 
bank accounts or credit cards)? 

0 

1 

2 

3 

4 

5 

7. BF How often do you read fiction (stories, 
novels)? 

0 

1 

2 

3 

4 

5 

8. BF How often do you read non-fiction (e.g., 
history, religious, self help books; reference 
materials; news sections of newspapers and 
current affairs websites)? 

0 

1 

2 

3 

4 

5 

9. BF How often do you use a library or visit a 
bookstore? 

0 

1 

2 

3 

4 

5 


Question 

Response 





The next questions are about activities at work. Please 
indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

disagree 

Disagree 

Neutral 

Agree 

Strongly 

agree 

10. BF 1 have the communication skills (i.e. ability to speak and 
listen to supervisors and co-workers) 1 need to do my main 
job well. 

1 

2 

3 

4 

5 

11. BF 1 have the math skills 1 need to do my main Job well. 

1 

2 

3 

4 

5 

12. BF 1 have the technicai skills 1 need to do my main job well. 

1 

2 

3 

4 

5 


100 




Measures of Success 
Final Report - March 2013 


B) Improvements in Job Performance and Attribution of Improvements to Training 

Suggested heading : "Your thoughts about how your work has changed since the training" 


Question 


These questions ask if you have changed the way you do 
specific parts of your job. 

13. F Since vou comoleted the course have vou noticed 

any improvement in your ability to get things done quicklv? 

1) No improvement SI 

2) A Slight Improvement 

3) Some Improvement 

4) A Big Improvement 

5) Not Applicable to My Job SI 

14. F SuDDOse vou had not Darticioated in the course. Do 

you think you would have achieved this improvement on 
your own? 

1) Very Unlikely 

2) Unlikely 

3) Not Sure 

4) Likely 

5) Very Likely 

15. F Since vou comoleted the course have vou noticed 

any improvement in your ability to complete tasks accurately, 
with few or no errors? 

1) No improvement SI 

2) A Slight Improvement 

3) Some Improvement 

4) A Big Improvement 

5) Not Applicable to My Job SI 
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Question 


16. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

17. F Since you completed the course have you noticed 

1) No improvement SI 

any improvement in your ability to be organized and plan 

2 ) A Slight Improvement 

your time effectively? 

3) Some Improvement 


4) A Big Improvement 


5) Not Applicable to My Job SI 

18. F Suppose you had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

19. F Since vou completed the course have vou noticed 

1) No improvement SI 

any improvement in your ability to understand and use 

2) A Slight Improvement 

workplace documents, such as workplace manuals or 

3) Some Improvement 

reports? 

4) A Big Improvement 


5) Not Applicable to My Job SI 
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Question 


20. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

21. F Since you completed the course have you noticed 

1) No improvement SI 

any improvement in your ability to be prepared for 

2) A Slight Improvement 

emergencies and follow safety procedures, such as handling an 

3) Some Improvement 

injury or fire in the workplace? 

4) A Big Improvement 


5) Not Applicable to My Job SI 

22. F Suppose you had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

23. F Since vou completed the course have vou noticed 

1) No improvement SI 

any improvement in your ability to use computers and work- 

2) A Slight Improvement 

related software programs effectively? 

3) Some Improvement 


4) A Big Improvement 


5) Not Applicable to My Job SI 
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Question 


24. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

25. F Since you completed the course have you noticed 

1) No improvement SI 

any improvement in your ability to use workplace equipment 

2) A Slight Improvement 

effectively, such as tools for your job or office machines? 

3) Some Improvement 


4) A Big Improvement 


5) Not Applicable to My Job SI 

26. F Suppose you had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

27. F Since vou completed the course have vou noticed 

1) No improvement SI 

any improvement in your ability to communicate and work 

2) A Slight Improvement 

effectively with your co-workers, such as completing different 

3) Some Improvement 

Job tasks in teams? 

4) A Big Improvement 


5) Not Applicable to My Job SI 
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Question 


28. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

29. F Since you completed the course have you noticed 

1) No improvement SI 

any improvement in your ability to communicate with 

2) A Slight Improvement 

customers, such as describing features and benefits of your 

3) Some Improvement 

company's products? 

4) A Big Improvement 


5) Not Applicable to My Job SI 

30. F Suppose you had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

31. F Since vou completed the course have vou noticed 

1) No improvement SI 

any improvement in your ability to respond effectively to 

2) A Slight Improvement 

customer questions or concerns, such as solving problems with 

3) Some Improvement 

an order? 

4) A Big Improvement 


5) Not Applicable to My Job SI 
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Question 


32. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 

33. F Since you completed the course have you noticed 

1) No improvement SI 

any improvement in your ability to encourage customers to 

2) A Slight Improvement 

buy your company's products and services? 

3) Some Improvement 


4) A Big Improvement 


5) Not Applicable to My Job SI 

34. F Suppose vou had not participated in the course. Do 

1) Very Unlikely 

you think you would have achieved this improvement on 

2) Unlikely 

your own? 

3) Not Sure 


4) Likely 


5) Very Likely 
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C) Participation in Education and Attitudes towards Continuous Learning 

Suggested heading : "Information about your education and training" 


35. BF Are you currently studying towards a degree, diploma or 
certificate? 

1) Yes - Please specify: 

2) No 

36. F Have you received any degrees, diplomas or certificates since you 
completed the course? 

1) Yes - Please specify: 

2) No 

37. F Have you participated in any other training since you completed 
the course? 

1) Yes - Please specify: 

2) No 

38. Please indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a) BF You are more likely to get a better job if you do some 
learning. 

1 

2 

3 

4 

5 

b) BF Learning new things makes you more confident. 

1 

2 

3 

4 

5 

c) BF Getting qualifications takes too much effort. 

1 

2 

3 

4 

5 
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D) Psychological Skills 

Suggested heading : "How you see yourself at work and in life" 


Question 


Please indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

39. BF 1 see myself as someone who has high self-esteem. 

1 

2 

3 

4 

5 

40. a) BF On the whole, 1 believe 1 do a good job 

1 

2 

3 

4 

5 

b) BF In my job I'm focused on learning and improving 
more than competing and being the best 

1 

2 

3 

4 

5 

c) BF 1 believe that what 1 do at work is important and 
useful 

1 

2 

3 

4 

5 

d) BF 1 try to plan out the things 1 have to do in my job 

1 

2 

3 

4 

5 

e) BF In my job, 1 use my time well and arrange my 
work area so that 1 can work under the best 
conditions 

1 

2 

3 

4 

5 

f) BF 1 persist in my job even when it is challenging or 
difficult 

1 

2 

3 

4 

5 

g) BF 1 get quite anxious in my job 

1 

2 

3 

4 

5 

h) BF If 1 work hard in my job it's usually to avoid 
failing or disapproval from my boss or colleagues 

1 

2 

3 

4 

5 

i) BF 1 don't think 1 have much control over how well 1 
do in my job 

1 

2 

3 

4 

5 


108 



Measures of Success 
Final Report - March 2013 


Question 








j) BF 1 find 1 sometimes reduce my chances of doing 
well in my job (e.g. waste time, not try hard, 
procrastinate) 

1 

2 

3 

4 

5 

k) BF 1 often feel like giving up in my job 

1 

2 

3 

4 

5 

Read each statement below and, as honestly as you can, 
answer the question: "How characteristic or true is this of 
you?" 

Very Untrue 

Somewhat 

Untrue 

Neutral 

Somewhat 

True 

Very True 

41. a) BF 1 make decisions on the spur of the moment (i.e. 

with little thought) 

1 

2 

3 

4 

5 

b) BF Meeting tomorrow's deadlines and doing other 
necessary work comes before tonight's play (e.g., 
before recreation or relaxation) 

1 

2 

3 

4 

5 

c) BF Generally, 1 am more focused on what is going on 
now than on what will happen in the future. 

1 

2 

3 

4 

5 

d) BF Since whatever will be, will be, it doesn't really 
matter what 1 do (i.e., 1 can't affect the future) 

1 

2 

3 

4 

5 

e) BF You can't really plan for the future because things 
change so much 

1 

2 

3 

4 

5 
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E) Social Networks, Inclusion, and Cohesion 

Suggested heading : "How you feel about your social life" 


Question 

Response 


42. During the last 12 months did you participate in any of the following 
groups or organizations? 

Yes 

No 

a) BF A political organization 

1 

2 

b) BF A sports team or recreation organization (e.g. 
baseball league, tennis club, curling club, etc.) 

1 

2 

c) BF A cultural, education or hobby group (e.g. theatre 
group, book club, bridge club, etc.) 

1 

2 

d) BF A service club (e.g. Lion's Club, Kiwanis, Knights of 
Columbus, Shriners, Kinsmen, etc.) 

1 

2 

e) BF A neighbourhood, civic, community or school 
association or group (e.g. parent teacher association. 
Ratepayers Association, community clubs, etc.) 

1 

2 

f) BF A group associated with a community of worship (e.g. 
a youth group associated with a church) 

1 

2 

g) BF International development organizations or other 
non-profit organizations 

1 

2 

h) BF Professional association or union 

1 

2 

i) BF Any other group or organization 

1 

2 

Question 

Response 


43. BF Do you participate in any workplace committees or groups 
such as a social committee, health and safety committee, 
employee-management committee, union committee or United 
Way committee? 

1) Yes 

2) No 
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The next few questions ask about the people you work and relax with - friends, family, colleagues, and other contacts - who you can go 
to for help and/or advice. Please note that there is a piece of paper provided. Before you attempt to answer the following questions, we 
recommend that you use this paper and take a few minutes to write down the names or initials of the people who would correspond to 
the various groups of contacts described in the questions below. You do not need to hand in this list - it is yours to keep or destroy. 


Question 

Response 

44. Circle your response to answer approximately how many 



individual contacts - including relatives, close friends or 



acquaintances - you have from whom you could easily: 



a) BF get help with household activities (such as child care, 

1) 

None 

household maintenance, household chores, personal care). 

2) 

One to Three 


3 ) 

Four to Six 


4 ) 

Seven to Ten 


5 ) 

More than 10 

b) BF get specialized advice (such as financial, medical or 

1) 

None 

legal advice). 

2) 

One to Three 


3 ) 

Four to Six 


4 ) 

Seven to Ten 


5 ) 

More than 10 

c) BF get emotional support (such as encouragement. 

1) 

None 

reassurance, confidential advice). 

2) 

One to Three 


3 ) 

Four to Six 


4 ) 

Seven to Ten 


5 ) 

More than 10 

d) BF get help with your job or career (assisting with your 

1) 

None 

current Job, or recommending you to a potential 

2) 

One to Three 

employer). 

3 ) 

Four to Six 


4 ) 

Seven to Ten 


5 ) 

More than 10 
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Question 

Response 

45. BF How many different people are there in questions 44 a, b, c 

1) 

None 

and d who you can call on for at least one of these kinds of help? 

2) 

One to Three 


3) 

Four to Six 


4) 

Seven to Ten 


5) 

More than 10 

46. BF How many of these contacts on your list would you say know 


1) All 

each other? 


2 ) Most 

3) Some 

4) Very few 

5) None 

47. BF How many of these contacts do you know from work - either 


1) All 

now or in the past? 


2 ) Most 

3) Some 

4) Very few 

5) None 

48. BF How many of these contacts would you say are working in 


1) All 

completely different occupations than you? 


2 ) Most 

3) Some 

4) Very few 

5) None 


Question Response 


Please circle your response for the following questions. 

49. If you lost your wallet or purse that contained $200 in it, how likely is 
it to be returned with the money in it if it was found by: 

Not at all 
Likely 

Somewhat 

Likely 

Very likely 

a) BF A neighbour who lives close by? 

1 

2 

3 

b) BF A clerk at a local grocery store? 

1 

2 

3 

c) BF A total stranger? 

1 

2 

3 
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F) Health and Well-Being Outcomes 

Suggested heading : "How you feel about your health, and satisfaction with your work and life" 


Question 


Response 


Please circle the appropriate number for each question 
below. 

Poor 

Fair 

Good 

Very 

Good 

Excellent 


50. BF In general, would you say your health is: 

1 

2 

3 

4 

5 

51. BF In general, would you say your mental health is: 

1 

2 

3 

4 

5 

Question 

Response 


52. Please indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a) BF My employer provides adequate facilities and flexibility 
for me to fit work in around my family life. 

1 

2 

3 

4 

5 

b) BF My line manager actively promotes flexible working 
hours / patterns. 

1 

2 

3 

4 

5 

c) BF 1 am satisfied with the overall quality of my working life. 

1 

2 

3 

4 

5 


53. BF Using a scale of 1 to 10 where 1 
means "Very Dissatisfied" and 10 
means "Very Satisfied", how do you feel 
about your life as a whole right now? 
Please circle your answer. 


Very Dissatisfied -» 


- » 


-» 


Very Satisfied 


10 
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G) Employment and Earnings 

Suggested heading : "Information about your employment" 


54. B When did you start working for this firm? 

Month: 

Year: 

55. BF What is your current job title? 

Job Title: 

56. BF When were you given your current Job title? 




Month: 



Year: 


57. BF How much do you usually earn before taxes and other deductions in 



this Job (that is, what are the gross earnings)? Please indicate the 

$ 

per (circle one): 

amount you earn in $ and the time period (e.g., per hour, week, etc.) 




1) 

Hour 


2) 

Day 


3) 

Week 


4) 

Two Weeks 


5) 

Month 


6) 

Year 


7) 

Other (please specify) 

58. BF On average, how many paid hours per week do you usually work at 



this Job? (Please include paid overtime hours if you usually work paid 


hours per week 

overtime.) 



59. BF Has there been a change in your usual number of paid hours per 

1) 

No change 

week at anytime in the past 6 months? 

2) 

Yes, a reduction to hours per week, for a period of weeks 


3) 

Yes, an increase to hours per week, for a period of weeks 
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60. BF How many weeks per month do you usually work at this job? 

1) 1 week 

2) 2 weeks 

3) 3 weeks 

4) 4 weeks 

61. BF Is this Job permanent, that is, has no specified end-date, or is it 
temporary? 

1) Permanent SI 

2) Temporary 

62. BF In what way is this job temporary? 

1) Work done through a temporary help agency or 
employment/staffing service firm 

2) Seasonal job 

3) Temporary, term, or contract (non-seasonal) 

4) Casual job 

5) Other (specify) : 

63. BF Over the past 6 months, have you experienced any layoffs or other 
periods of work stoppage at your job? 

1) No 

2) Yes (please specify) 

periods of work stoppage, for a total of weeks 

64. BF Are you currently working in a second job or business? 

1) Yes 

2) NoS4 

65. BF What is your job title in your second job? 

Job Title: 

66. BF How much do you usually earn before taxes and other deductions at 
your second job (that is. what are your gross earnings)? Please indicate 
the amount you earn in $ and the time period (e.g., per hour, week, 
etc.) 

$ . per (circle one): 

1) Hour 

2) Day 

3) Week 

4) Two Weeks 

5) Month 

6) Year 

7) Other (please specify) 
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67. BF On average, how many paid hours per week do you usually work at 
your second job? (Please include paid overtime hours if you usually 
worked paid overtime.) 

hours per week 

68. BF How many weeks per month do you usually work at that second job? 

Circle your response: 

1) 1 week 

2) 2 weeks 

3) 3 weeks 

4) 4 weeks 

69. F Since vou completed the course, have vou had an increase in 

wages or earnings? 

1) No SI 

2) Yes, an increase in earnings for mv main iob 

3) Yes, an increase in earnings for mv second iob 

4) Yes, an increase in earnings for both mv main iob and mv second iob 

70. F SuDDOse vou had not oarticioated in the course. Do vou think 

your earnings would have increased as much as they did? 

1) Very Unlikely 

2) Unlikely 

3) Not Sure 

4) Likely 

5) Very Likely 

71. F Since vou completed the course, have vou received a iob 

promotion? 

1) No SI 

2) Yes, a promotion in mv main iob 

3) Yes, a promotion in mv second iob 

4) Yes, a promotion in both mv main iob and mv second iob 

72. F Suppose vou had not participated in the course. Do vou think 

you would have received this promotion? 

1) Very Unlikely 

2) Unlikely 

3) Not Sure 

4) Likely 

5) Very Likely 
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H) Training Feedback: Course Quality and Relevance 

Suggested heading : "Your thoughts about the training" 


Question 

Response 





73. Please indicate whether you agree/disagree with each of the 
following statements: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a) 

F The purpose of the course was clearly explained 

1 

2 

3 

4 

5 

b) 

F The course purpose was met 

1 

2 

3 

4 

5 

c) 

F Facilitator was knowledgeable about the subject matter 

1 

2 

3 

4 

5 

d) 

F Facilitator presented concepts and material clearly in a 
way that helped me understand 

1 

2 

3 

4 

5 

e) 

F Facilitator promoted discussion and participation 

1 

2 

3 

4 

5 

f) 

F Facilitator kept discussion and activities on track 

1 

2 

3 

4 

5 

g) 

F The training was relevant to my Job 

1 

2 

3 

4 

5 

h) 

F 1 intend to apply what 1 learned during the training to my 
Job 

1 

2 

3 

4 

5 

i) 

F There are ways for me to practice my new skills on the 
Job. 

1 

2 

3 

4 

5 

j) 

F 1 would recommend the training program to my co- 
workers, or other employees in similar Jobs 

1 

2 

3 

4 

5 
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Question 

Response 

74. F About what percent of the sessions did you attend? 

Enter percent attended 

75. F Please share any information you believe would help us to 
improve this course. 

Enter comments here: 

76. F What is your overall rating for this course? 

1) Poor 2) Fair 3) Good 4) Very Good 5) Excellent 


I) Training Attitudes and Goals 

Suggested heading : "How you feel about the training course you are about to start" 


Question 

Response 





77. Please indicate whether you agree/disagree with each of 
the following statements: 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

a) B 1 am looking forward to taking this training. 

1 

2 

3 

4 

5 

b) B 1 want to learn more about this topic. 

1 

2 

3 

4 

5 

c) B 1 am motivated to do the best 1 can in this 
course. 

1 

2 

3 

4 

5 

d) B My supervisor supports me in taking this 
course. 

1 

2 

3 

4 

5 

e) B 1 am just taking this training for the credit. 

1 

2 

3 

4 

5 
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Please circle all that apply to your situation: 

1) Job Requirement 

2) To help me do my job better 

3) To advance my career 

4) Required for further education 

5) To improve my oral communication skills 

6) To improve my problem-solving skills 

7) To improve my math skills 

8) To improve my reading skills 

9) To improve my writing skills 

10) Other (specify) 
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J) Demographics & Household Characteristics 

Suggested heading : "Information about you and your family" 


Question 

Response 


79. B Gender 

1) Female 

2) Male 

80. B Date of birth 

Month Year 





81. B Marital status 

1) Married 

2) Common law 

3) Single never married 

4) Separated 

5) Divorced 

6) Widowed 

82. B How many adults 18 years of age or older, including yourself, live in 
your household? Please include all adults who usually live with you, 
including those who may be away attending school, travelling or in 
hospital. 

Number of adults: 


83. B How many children under the age of 18 live in your household? Please 
Include all children who usually live with you, including those who may 
be away attending school, travelling or in hospital. 

Number of children under 18: 


84. B How many of these children are under the age of 6? 

Number of children under 6: 
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85. B What was vour household income in the last 12 months, before taxes 
and other deductions? 

Include all sources (for example, wages, alimony, investments. 
Employment Insurance, social assistance, grants). Please give us your 
best guess if the exact figure is not known. 

1) Less than $10,000 

2) $10,000 to less than $20,000 

3) $20,000 to less than $30,000 

4) $30,000 to less than $40,000 

5 ) $40,000 to less than $50,000 

6 ) $50,000 to less than $60,000 

7) $60,000 to less than $70,000 

8) $70,000 to less than $80,000 

9) $80,000 to less than $90,000 

10) Greater than $90,000 

86. B In what country were you born? 

1) Canada SI 

2) Other (please specify): 

87. B If you were born outside Canada, in what year did you come to live in 
Canada? 

Year: 

88. B Are you an Aboriginal person, that is. First Nations (North American 
Indian), Metis or Inuk (Inuit)? 

1) No, not an Aboriginal Person 
2 ) Yes, First Nations 

3) Yes, Metis 

4) Yes, Inuit 

5) Yes, Other (please specify) 

89. B What language do you speak most often at home? 

1) English 

2) French 

3) Other (please specify): 
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Question Response 


90. B Do you speak any other languages on a regular basis at home? 

1) No 

2) Yes, English 

3) Yes, French 

4) Yes, Other (please specify): 

91. B What is the language that you first learned at home in childhood and 
still understand? 

1) English 

2) French 

3) Other (please specify): 
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K) Previous Education and Training Experience 

Suggested heading : Incorporate into Module C "Information about your education and training" 


92. B What is the highest level of education you have completed? 

1) Less than a high school diploma 

2) A high school diploma or equivalent 

3) Trade/Vocational or Apprenticeship diploma or certificate 

4) Community college or CEGEP diploma or certificate 

5) University degree 

6) Any other diploma, degree or certificate(please specify) 

93. B Think about your overall experience of school from first grade to when you 
left. Would you say your experience was: 

1) very negative 

2) somewhat negative 

3) neutral 

4) somewhat positive 

5) very positive 

94. B Have you participated in any training over the past 6 months? 

1) Yes - Please specifv: 

2) No 
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L) Health/Disability Status 

Suggested heading : Incorporate into Module F - "How you feel about your health, and satisfaction with your work and life" 


Question 

Response 



Often 

Sometimes 

Not at All 

95. B Do you have any difficulty hearing, seeing, 

communicating, walking, climbing stairs, bending, 
learning or doing any similar activities? 

1 

2 

3 

96. B Does a physical or mental condition or health 

problem reduce the amount or kind of activity you can 
do at work? 

1 

2 

3 

97. B What about during non-work related tasks, including 
at school, home, or leisure? 

1 

2 

3 
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Appendix B.l 

Guidelines for Using the Question Inventory: 

Interviews with Employers and Workplace Educators 

In the Measures of Success project, employer and instructor interviews facilitated three major research 
goals: 


• They captured post-training changes in key business outcomes, as well as the 
attribution of these changes to training. 

• They captured key business contextual measures, allowing us to establish which of 
these measures acted as mediating factors, i.e. which either constrained or supported 
post-training gains among individual learners as well as post-training improvements in 
business outcomes at the firm level. 

• They captured key training implementation factors, allowing us to establish which of 
these measures influenced training outcomes for participants and businesses 

The attached employer/instructor inventory includes recommended questions to address these 
research goals. A brief guide to the inventory is provided below. 

I. Business Outcomes 

It is important to understand what post-training outcomes employers observe for their businesses, and 
to what extent they attribute any changes in business outcomes to the training. While the 
recommended inventory questions may be subjective, they are likely the best way to gather information 
on business outcomes in the absence of a control group of similar employers who had not participated 
in the training and whose outcomes could therefore be considered the counterfactual^^. 

The business outcomes that employers could potentially be asked about are: productivity, controlling 
costs and reducing errors, product/service quality, customer service, sales, turnover, absenteeism, 
health and safety, employee-management relations, learning culture, sense of unity/belonging, and 
employee involvement/participation. Note that it would not be necessary to ask all employers about the 
full set of business outcomes. Some outcomes may not apply in some workplaces, for example, 
increased sales revenues and health and safety may not be sought after goals of some non-profit 
organizations. 

The recommended questions for each potential business outcome are presented in the attached 
inventory. For each outcome, employers should be first asked whether it changed over the six months 
since the training and whether or not they could supply information to support this. Then, for outcomes 
that changed, employers should be asked their impression of the likelihood that the change would have 
happened if the training hadn't taken place. In the absence of a control group of similar employers who 


12 


That is, outcomes the participating employers would have experienced if their employees hadn't received the training. 
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had not participated in the training, this line of questioning will provide the best approximation of 
outcomes the participating empioyers wouid have experienced if their empioyees hadn't received the 
training. Employers who respond with "likely" or "very likely" should be asked, in open-ended fashion, 
to specify how the training made a difference, which will further support and elaborate the response. 
Those who respond with "unlikely" or "very unlikely" or "unsure" should be asked to suggest other 
factors that may have contributed to the change. 

II. Business Contextual Factors 

Research indicates that the role Essential Skills training plays in the attainment of particular business 
and learner outcomes may be affected by (or mediated by) a number of factors tied to the business 
context and the way that training is implemented. These factors may be indicators of likely training 
success or failure and can help identify firms that may be at risk of not attaining expected outcomes. 

This can help workplace educators customize suitable training supports as well as managing employer's 
expectations. In the presence of significant resource constraints, they could also be used by training 
administrators to guide targeting and selection of businesses to allocate scarce training resources to 
places where they have the best chance of success. 

Recommended business context measures are described below. 

a. Basic aspects of the firm relating to its sector and size - Section B, Questions 1 & 2 

Business size and sector may influence training outcomes. The Measures of Success study used a 
two-category measure for sector (goods or services), and a three-category size measure for size 
(smaii: less than 100 employees; medium: 100-499 employees; or iarge: 500 or more employees). In 
the Measures of Success study, businesses in services and with a large (500 or more) number of 
employees were more likely to exhibit post-training improvements in job performance - however, it 
is unclear whether this pattern of results would be replicated in other studies. 

b. Presence of performance incentives - Section B, Question 3 

In the Measures of Success study, the presence of performance incentives was a yes or no measure: 
yes, if there was remuneration in the form of a bonus or salary increase for superior performance on 
the job; no if not. The analysis found that employees in businesses that offered performance 
incentives were more likely to experience post-training gains in some skills, confidence and other 
outcomes. 

c. Amount of existing training- Section B, Question 4 

Our research indicated that improvements in business outcomes and employee job performance 
were more likely when Essential Skills training was implemented in workplaces with greater levels of 
existing training resources, compared to workplaces with fewer training resources. Information on 
existing training resources was captured in question 4 of Part B, which yields qualitative open-ended 
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responses that must be interpreted with care. Based on responses to this question, we categorized 
businesses in the Measures of Success project as: 

High (where words such as "all", "widespread" and "significant" were used) 

Medium/high (indication of some training other than just supervisor or peer led) 

Low/medium (indication of just training that is supervisor provided or consists of job shadowing 
or peer mentoring), or 

Low (where words such as limited" or "only general" were used). 

Owing to the imprecise nature of this question, a possible alternative is suggested: asking employers 
to indicate how much they spent on training over the previous 12 months. These responses could be 
scaled by the number of employees at the workplace and used to categorize participating 
businesses in a more rigorous fashion than the question that was used in the Measures of Success 
project. 

III. Training Implementation Factors 

For this set of indicators, the focus is on implementation factors associated with the training. Our results 
revealed that implementation factors may play an important role in the likelihood of attaining successful 
post-training business and learner outcomes. Questions to collect information on implementation 
factors should be asked of both employers and workplace educators. 

a. Basic characteristics of the training- Section C, Questions 1 to4 

In the Measures of Success study, three basic measures of training delivery in particular were 
indicative of success: 

• Prior experience (question 1): The success of the Essential Skills (ES) training may be influenced 
by whether or not the business had prior experience with the provincial department for the 
delivery of this kind of training. In Measures of Success, those with a prior relationship with the 
department to deliver ES training experienced greater improvements in outcomes in the short- 
term (i.e. three months after training). However, in the longer-term (six months after training), 
improvements were just as likely for firms that had no prior relationship with the department 
with respect to ES training delivery. 

• Dosage (questions 2 & 3): In Measures of Success, learners who received greater amounts of ES 
training (at least 15-20 hours) achieved greater improvement in skills, confidence and job 
performance than those with otherwise similar characteristics who received lower amounts of 
training (less than 10 hours). 
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• Location (question 4): Whether or not the ES training takes place onsite (i.e. the site where the 
learners normally work) or offsite may have an impact on participant outcomes. In Measures of 
Success, onsite training was associated with greater success for participants in terms of skills 
and confidence gains. 

b. Training-Business Alignment - Section C, Questions 5 to 11 

In Measures of Success, ES training that was well aligned to the needs of the business and its 
employees was one of the most powerful predictors of job performance gains. A series of seven 
questions (Section C - questions 5 to 11) was used to measure training-business alignment. It is 
important that, when asking these questions, the investigator encourages the employer and 
instructor to answer in job performance and behavioural terms, with a focus on employees who will 
be participating in the training. Note that, as the questions seek open-ended responses, this is not a 
straightforward exercise and necessitates judgment when coding the responses. What we present 
below is meant as guidelines only. 

In order to categorize businesses according to different levels of training-business alignment, six 
alignment sub-factors, listed below, were used: 

• How well employer can articulate the training need in Job-performance terms: It is likely that 
having a project sponsor with at least some understanding of the "end goal" of the training in 
terms of employee performance should increase the likelihood that the training will have a 
positive effect on business results. This can be measured using questions 5 and 6 of Section C. A 
suggested coding of the responses would be: i) yes, if employers identify at least one specific 
performance issue such as the need to reduce errors in reading blueprints; ii) somewhat, if they 
identify only a general performance issue such as a need to improve communication 
across/within departments; or iii) no, if they did not identify any performance issues at all even 
in general terms. 

• How well the Instructor can describe the employees' Job-performance needs: Similarly, 
workplace educators who have a more detailed understanding of the specific things that 
employees should be doing more of, less of or differently would deliver training that is likely to 
be more effective in improving performance. This concept is captured by questions 7 and 8 of 
Section C. Responses to these questions can be coded to make up three categories: i) yes, if 
workplace educators give a specific example of what employees need to do more or less of or 
differently, or they describe the process they would take to identify performance needs; ii) 
somewhat, if they describe performance needs in relatively general terms and do not indicate 
that they would take steps to develop a more specific understanding of the gap between 
desired versus actual behavior; or iii) no, if they do not describe any specific on-the-job 
performance needs. 
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• How well the Instructor can describe the business needs motivating the training: The source 
questions are Questions 7 and 8 of Section C, where the instructor is asked to indicate what the 
employer's motivation is for the training and what the employer would want weaker employees 
to do better on the job. Responses are coded as yes or no, based on whether or not the 
responses suggest that the instructor was able to articulate a business need underlying the 
firm's participation in the training. 

• Degree to which Instructor indicates that the curriculum was customized to reflect: 

Specific job performance needs (Section Q question 10) - yes or no 
Workplace materials and scenarios (Section C, question 11) ) - yes or no, and 
Employee skill assessments/levels (Section C, question 12) - yes or no 

In Measures of Success, alignment was categorized into one of three levels, depending how many 
"yeses" were obtained for the six sub-factors of alignment described above. 

i) If for a business, most sub-factors had a "yes", then alignment was rated as well aligned. 

ii) If most did not have a "no", i.e., there was a mixture of "yes", "somewhat", and "no", then 
alignment was judged to be moderate. 

ill) If the majority of the sub-factors were measured as "no" for that business, then alignment was 
deemed to be possibly at risk. 

Alternatively, information sufficient to determine the degree of training-business alignment may be 
found in the instructor's course curriculum. If the curriculum is provided, it may not be necessary to ask 
Questions 7 to 11 in Section C. 
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Appendix B.2 

Question Inventory for 

Interviews with Employers and Workplace Educators 


A comprehensive inventory of recommended questions, to allow stakeholders to construct 
measurement instruments for employer or instructor interviews, is presented below. 

Questions are organized into three sections: A) Business Outcomes, B) Business Contextual Factors, and 
C) Training Implementation Factors. Beside each section heading is an indicator as to when the 
questions in that section should be asked: 

BASELINE ONLY; pre-training only 
FOLLOW-UP ONLY; post-training only 

An accompanying Guide with descriptions of key measures is attached to help stakeholders prioritize 
and make decisions about which questions to use. 

A) Business Outcomes - FOLLOW-UP ONLY 
Productivity 


1. Over the last six months (since the training began), do you think that productivity (completing 
tasks quickly and completely) among employees has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

''Don't Know 

^N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

2. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

3. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

4. Can you be specific about how you think the training made a difference? 
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IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

5. What other factors besides training contributed to the change? 


Costs and Errors 


6. Now I would like to ask you about controlling costs and reducing errors . Over the past 6 months 
(since the training began), do you think that costs and errors have...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

7. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

8. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

9. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

10. What other factors besides training contributed to the change? 


Product/Service Quality 


11. Now I would like to ask you about product/service quality . Over the past 6 months (since the 
training began), do you think that product/service quality has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 
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IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

12. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 


13. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

14. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

15. What other factors besides training contributed to the change? 


Customer Service 


16. Now I would like to ask you about customer service . Over the past 6 months (since the training 
began), do you think that customer service has...? 


^ Improved 

^Worsened 

^Neither improved 

''Don't Know 

®N/A for firm 



nor worsened 




IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

17. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

18. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

19. Can you be specific about how you think the training made a difference? 
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IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

20. What other factors besides training contributed to the change? 


Sales 

21. Now I would like to ask you about sales . Over the past 6 months (since the training began), do you 
think that sales have...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

22. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

23. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

24. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

25. What other factors besides training contributed to the change? 


Turnover 


26. Now I would like to ask you about employee turnover . Over the past 6 months (since the training 
began), do you think the number of staff leaving this location has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 
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IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

27. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 


28. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

29. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

30. What other factors besides training contributed to the change? 


Absenteeism 


31. Now I would like to ask you about employee absenteeism . Over the past 6 months (since the 
training began), do you think the rate of absenteeism among employees has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

32. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

33. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

34. Can you be specific about how you think the training made a difference? 
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IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

35. What other factors besides training contributed to the change? 


Health and Safety 


36. Now I would like to ask you about workplace health and safety . Over the past 6 months (since the 
training began), do you think the number of health and safety issues have...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

37. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

38. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

39. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

40. What other factors besides training contributed to the change? 


Employee-Management Relations 


41. Now I would like to ask you about employee-management relations . Over the past 6 months 
(since the training began), do you think that employee-management relations have...? 


^ Improved 

^Worsened 

^Neither improved 

^Don't Know 

®N/A for firm 



nor worsened 
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IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

42. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 


43. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

44. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

What other factors besides training contributed to the change? 


Learning Culture in the Workplace 


45. Now I would like to ask you about learning culture in the workplace . Over the past 6 months 

(since the training began), do you think that voluntary participation in training among staff has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 






46. Over the past 6 months (since the training began), do you think that certification rates (recognized 
outside of the workplace) among staff have...? 

^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


47. Over the past 6 months (since the training began), do you think that mentoring and knowledge 
exchange among staff has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 
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IF RESPONDENT INDICATED THAT A CHANGE OCCURRED IN ANY OF THESE AREAS: 

48. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 


49. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

50. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

What other factors besides training contributed to the change? 


Sense of Unity and Belonging 

51. Now I would like to ask you about employees' sense of unity and belonging . Over the past 6 
months (since the training began), do you think that feelings of unity and belonging among staff 
have...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

52. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

53. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

54. Can you be specific about how you think the training made a difference? 
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IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

55. What other factors besides training contributed to the change? 


Employee Involvement and Participation 


56. Now I would like to ask you about worker involvement and participation . Over the past 6 months 
(since the training began), do you think that the frequency of worker involvement and 
participation has...? 


^ Increased 

^Decreased 

^Neither increased 
nor decreased 

^Don't Know 

®N/A for firm 


IF RESPONDENT INDICATED THAT A CHANGE OCCURRED: 

57. Do you have an updated report that shows this change? If yes, would you be willing to share it 
with us? 

58. Do you think the change would have happened if the training hadn't taken place? 


Very 

unlikely 

Unlikely 

Not sure 

Likely 

Very 

Likely 

1 

2 

3 

4 

5 


IF RESPONDENT SAID VERY UNLIKLEY OR UNLIKELY: 

59. Can you be specific about how you think the training made a difference? 
IF RESPONDENT SAID NOT SURE, LIKELY, OR VERY LIKELY: 

60. What other factors besides training contributed to the change? 
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B) Business Contextual Variables -BASELINE ONLY 

The following questions are recommended for employer surveys (E). 

a. Firm Size and Sector 

1. In what industry is the firm? 

2. How many employees work at this worksite? (E) 

b. Presence of Performance Incentives 

3. Is there is reward system for good job performance? If yes, please briefly describe this system, 
including the various types of rewards that employees may receive (e.g. increased pay). (E) 

c. Amount of Existing Training 

4. How do employees get trained? How do they learn what is expected of them? (E) 

Alternative: Over the last 12 months, what was the approximate amount your business spent 
on training? 


C) Training Implementation Variables -BASELINE ONLY 

The following questions are recommended for employer sign-up sheets (E sign-up sheet), employer 
surveys (E), or training instructor surveys (I). 

a. Basic Characteristics of Essential Skills Training Delivered 

Prior experience with Essential Skills/department 

1. Previous experience with Essential Skills through department (E sign-up sheet) 

Dosage: length of the training 

2. Number of hours (E sign-up sheet, confirmed by I below) 

3. How is the course scheduled: number of classes and length of each class? (I) 

Location: onsite versus offsite delivery: 

4. Where is the course held? (I or E sign-up sheet) 
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b. Alignment of Training to Business Needs 

Degree to which employer clearly describes need in job-performance terms 

5. What is your motivation for this training initiative? What are you hoping employees will do 
differently after the training? (E) 

Probe: Can you give me examples of issues that prompted the training? (Probe for specific 
information about performance issues, such as types of mistakes, incomplete tasks, etc.) 

6. What does a "star" employee look like in terms of their job performance? What does a weaker 
employee look like? (Focus on employees participating in training.) (E) 

Probe: What would you like the weaker performers to do that they are not currently doing? 
(Encourage the respondent to provide examples. Ask the person to describe what employees should 
be doing in behavioural terms .) 

Degree to which instructor describes needs in job-performance terms, and 

Degree to which instructor clearly identifies a business need that motivated the training 

7. What do you think is the employer's motivation for this training initiative? What are they hoping 
employees will do differently after the training? (I) 

Probe: Can you give me examples of issues that prompted the training? (Probe for specific 
information about performance issues, such as types of mistakes, incomplete tasks, etc.) 

8. What would the employer like weaker performers to do that they are not currently doing? What 
would the employer say are the essential skills that employees may need to strengthen in order to 
become exemplary ("star") employees? (Focus on employees participating in training.) (I) 

Probe: Encourage the respondent to provide examples. Ask the person to describe what employees 
should be doing In behavioural terms .) 

Degree to which instructor indicates customization of curriculum to: job-performance needs, workplace 
materials and scenarios, and employee skill levels 

9. How did you customize the curriculum for this worksite? (I) 

10. Are you able to use actual workplace materials in the curriculum? If yes, please provide some 
examples. (I) 

11. How did the results of the initial skills assessments influence the training curriculum? (I) 

Alternatively, information about the instructor's identification of job performance/business needs 
may be found in the course curriculum, materials, and/or skills assessment instruments. If these are 
provided, it may not be necessary to ask questions #7-11 . 
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